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Accurate  and  fast  retrieval  and  efficient  storage  are  two  principal  needs  in  an  image 
database.  Image  representation  is  the  crucial  factor  in  fulfilling  these  requirements.  This 
research  proposes  an  image  representation,  including  an  image  data  structure  and  a  spatial 
knowledge  representation,  that  will  facilitate  the  above  two  demands.  The  proposed  data 
structure,  called  a  line  string,  encodes  an  image  via  a  string  of  linear  skeleton.  The 
proposed  spatial  representation,  called  an  object-oriented  line  string,  provides  a  structural 
and  easy  way  to  specify  the  line  string  in  a  database  query.  The  research  shows  that 
the  line  string  exhibits  advantages  over  other  structures,  but  may  require  more  storage 
and  retrieval  time. 

Using  the  line-string  representation,  image  retrieval  becomes  line-string  matching, 
i.e.,  finding  the  longest  approximate  common  subsequence  (LACS)  of  two  strings.  The 
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LACS  problem  is  divided  into  smaller  parts  in  order  to  lessen  its  complexity.  Some 
of  these  parts  have  been  proven  to  be  NP-hard.  An  approximation  algorithm  and 
an  optimization  neural  network  are  constructed  to  find  a  near-optimal  solution  for  the 
problem.  A  ratio  bound  of  the  approximation  algorithm  and  a  comparative  performance 
of  the  two  methods  also  are  discussed. 

To  realize  the  advantages  of  the  proposed  research,  a  prototype  traffic 
sign  retrieval  and  browsing  system  is  available  on  the  World  Wide  Web,  at 
http://www.cise.ufl.edur wenchen/idb.  The  experiment  results  show  any  image  in 
the  system  can  be  precisely  retrieved  as  long  as  each  image  has  a  different  image  string. 
The  accuracy  of  image  retrieval  depends  on  the  query  details. 


IX 


CHAPTER  1 
INTRODUCTION 

The  core  of  an  image  database  system  (IDBS)  consists  of  an  image  store  and 
a  relational  text  database.  The  image  store  caches  raw  images.  The  features  of  raw 
images  are  extracted  and  stored  in  the  relational  database  according  to  the  structure  of  the 
database.  A  successful  IDBS  has  to  furnish  (i)  accurate  and  fast  retrieval  and  (ii)  efficient 
storage.  Image  representation — namely,  data  structure  and  spatial  representation — is 
crucial  to  achieving  an  image  database  that  meets  the  above  two  demands.  Various 
image  representations  have  been  proposed  in  the  literature  [8,  9,  10,  13,  33],  but 
string  representations  perform  best  in  storage  and  retrieval.  String  image  representations 
soon  may  become  the  dominate  type  used  in  databases;  nevertheless,  they  lack  several 
requirements  of  databases.  The  following  weak  points  are  typical  in  string  data  structures: 

1.  String  image  data  structures  for  symbolic  objects  portray  the  spatial  relations  of 
centroids  or  extreme  points  of  objects,  rather  than  the  actual  relative  positions 
of  objects. 

2.  Symbolic  object  representation  falls  short  of  characterizing  an  object's  structural 
shape  and  size. 

3.  Some  objects  can  be  detected  but  cannot  be  recognized.  In  these  cases,  im- 
age retrieval  depends  on  the  objects'  configurations,  a  requirement  generally 
unachievable  with  symbolic  objects. 

1.1  A  Line-String  Image  Representation 

Like  other  image  representations,  however,  string  image  representations  fail  to 
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satisfy  fully  the  requirements  of  image  databases.  They  particularly  fall  short  in  describing 
shapes,  sizes,  and  relative  positions  of  image  objects.  This  paper  proposes  a  new  string 
image  representation  to  facilitate  image  storage  and  retrieval.  It  includes  a  data  structure, 
called  a  line  string,  and  an  object-oriented  line  string  to  represent  spatial  knowledge, 
called  an  OOLS. 

A  linear  approximation  algorithm  converts  the  image  skeleton  into  line  segments. 
The  endpoints,  relative  lengths,  and  relative  angles  of  these  straight  lines  are  then  stored 
in  a  line-string  data  structure.  A  line  string  maintains  line  segments  in  a  string.  Lines 
or  points  are  described  in  polar  coordinate  systems.  Each  line  or  point  in  the  plane  is 
assigned  coordinates  (r,  9),  where  r  is  the  length  and  9  is  the  measure  of  an  angle  with 
an  initial  side  corresponding  to  the  polar  axis  and  a  terminal  side  coinciding  with  the  line 
or  point.  Image  retrieval  then  becomes  line-string  matching. 

An  OOLS  provides  a  more  structural  and  lucid  way  to  realize  the  line  string  in  a 
database  query.  A  query  specifies  the  spatial  relationship  through  a  line  string,  an  OOLS, 
a  linear  image,  a  skeletal  image,  or  a  segmented  image.  An  OOLS  also  can  reconstruct  a 
linear  image.  Because  an  OOLS  contains  more  information  on  configurations  and  relative 
positions  of  image  objects  than  other  string  image  representations,  image  reconstruction 
using  the  OOLS  method  generally  is  more  accurate. 

The  number  of  spatial  relations  possessed  by  a  line  string  is  given  by  a  theorem 
constructed  step-by-step  by  other  lemmas  and  theorems.  The  space  needs  of  a  line  string 
are  discussed,  as  are  the  comparisons  of  storage  space  and  matching  speed  between  line 
strings  and  other  string  data  structures.  It  shows  that  a  line  string  exhibits  some  advantages 
over  other  string  image  data  structures  but  may  require  more  storage  and  retrieval  time. 
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1 .2  A  Line-String  Matching  Method 

Using  the  line-string  representation,  image  retrieval  becomes  line-string  matching, 
i.e.,  finding  the  longest  common  subsequence  (abbreviated  LCS)  between  two  images. 
However,  a  longest  common  subsequence  does  not  always  reveal  the  degree  of  difference 
between  two  strings  required  for  this  project.  The  problem  is  generalized  to  a  longest 
approximate  common  subsequence  (abbreviated  LACS)  problem,  which  is  finding  a 
maximum-gain  approximate  common  subsequence  of  two  strings.  An  approximate 
subsequence  of  a  string  X  is  a  string  edited  from  a  subsequence  of  X.  String  Z  is 
an  approximate  common  subsequence  of  two  strings  X  and  y  if  Z  is  an  approximate 
subsequence  of  both  X  and  Y.  The  gain  function  g  assigns  a  nonnegative  real  number 
to  each  subsequence.  The  problem  is  divided  into  smaller  parts  in  order  to  lessen  its 
complexity.  Some  of  these  parts  have  been  proven  to  be  NP-hard.  An  approximation 
algorithm  and  an  optimization  neural  network  are  constructed  to  find  a  near-optimal 
solution  for  the  problem. 

An  approximation  algorithm  finds  an  approximate  LACS  (abbreviated  ALACS)  of 
two  strings.  The  procedure  APPROX_LACS  repeatedly  calls  an  LCS-routine,  beginning 
with  allowing  zero  number  of  edit  operations  for  each  selected  symbol.  Each  round, 
the  allowable  number  of  edit  operations  is  increased  by  one.  The  symbols  selected  are 
marked  off  from  the  input  strings  to  prevent  consideration  in  the  next  round.  A  ratio 
bound  of  the  approximation  algorithm  is  also  discussed.  A  modified  Hopfield  neural 
network  is  designed  to  solve  the  LACS  problem.  A  technique  of  interception  is  used  to 
determine  the  values  of  network  coefficients.  The  comparative  performance  of  the  two 
methods  and  some  possible  LACS  applications  are  discussed. 
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1.3  An  Image  Retrieval  System 

To  realize  the  advantages  of  the  proposed  research,  a  prototype  traffic 
sign  retrieval  and  browsing  system  is  available  on  the  World  Wide  Web,  at 
http://www.cise.ufl.edur wenchen/idb.  The  system  interface  provides  the  following 
choices: 

1.  Polar  or  rectangular  coordinate  system; 

2.  LCS,  LACS,  ALACS,  or  Hopfield  neural  network  search  method;  and 

3.  Centroid,  endpoint,  or  line  string. 

The  length  and  angle  data,  given  by  the  polar  system,  and  the  x  and  y  coordinates 
data,  given  by  the  rectangular  system,  supply  different  and  useful  information.  It  is  an 
advantage  for  the  system  to  provide  both  coordinate  systems.  The  experiment  results 
show  the  ALACS  method  outperforms  the  LCS  method.  Any  image  in  the  system  can 
be  precisely  retrieved  as  long  as  each  image  has  a  different  image  string.  The  accuracy 
of  image  retrieval  depends  on  the  query  details. 


CHAPTER  2 
RELATED  RESEARCH 

The  next  generation  of  active  image  information  systems  should  be  designed  based 
upon  the  notions  of  generahzed  icons  and  active  indexes,  resulting  in  smart  images.  [7] 
A  generalized  icon  is  an  object  consisting  of  a  pair  made  of  the  image  (physical  part) 
and  its  logical  interpretation  (logical  part).  If  image  indexing  methods  are  active,  partial, 
dynamic,  visible,  and  imprecise,  they  are  active  indexes.  To  reach  the  above  notions 
is  the  goal  of  the  proposed  image  system.  The  line-string  image  representation,  which 
consists  of  a  pair  of  an  image  and  its  image  strings,  is  a  generalized  icon.  The  LACS 
search  method  has  not  fully  met  the  characteristics  of  active  indexes  yet. 

2.1  Related  String  Image  Representations 

Chang  and  his  colleagues  [10,  11]  suggested  a  2-D  string  that  maintains  the  objects' 
spatial  knowledge  of  their  location  within  images.  Images  are  stored  through  logical 
pictures  retained  in  2-D  strings.  A  retrieval  query  may  specify  a  2-D  string,  transforming 
retrieval  into  a  2-D  subsequence  matching.  Figure  1  exhibits  a  symbolized  image  /,  with 


Figure  1.  A  symbolic  image  /  for  2-D  string. 


each  symbol  corresponding  to  an  object.  We  shall  briefly  discuss  how  to  create  the  2-D 
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String  from  an  image.  Let  V  =  {Oj,  O2,  . . .  ,  0^}  he  a  set  of  symbols,  where  each  0„ 
i  =  1,  2,  . .  .  ,  n  represents  a  pictorial  object.  Let  A  =  {'=',  '<',  ':'}  denote  a  set  of 
relations,  where  each  element  in  A  represents  the  spatial  relation  between  two  objects. 
Here  '='  represents  the  spatial  relation  "at  the  same  x  or  y  location  as,"  '<'  represents 
the  spatial  relation  "to  the  west  of  or  the  south  of,"  and  ':'  represents  the  relation  "in 
the  same  set  as."  A  2-D  string  over  V  U  A  is  represented  as  (0/  r/  O2  .  .  .  r^.i"  0„,  Opd) 
n^'  Op(2)  .  .  .  r„./  Opf„,),  where  r2^  .  . .  ,  r„.,\  r/,  . . .  ,  r„./  are  relations  from  the 
set  A  and  p  is  a  permutation  function  on  { 7,  2,  ...,«} .  For  instance,  the  2-D  string 
of  Figure  1  is  given  {a  <  a  =  h  <  c,  a  =  a  <  c  <  h).  Lee  et  al.  [36]  defined  a 
similarity  measure  based  on  a  2-D  string's  LCS.  The  algorithm  for  similarity  retrieval 
is  to  transform  the  2-D  string  LCS  problem  to  the  maximal  complete  subgraph  (clique) 
problem.  Lee  and  Shan  [35]  proposed  another  efficient  methods  for  retrieval  by  objects, 
retrieval  by  pairwise  spatial  relationships  and  retrieval  by  subpicture.  All  of  the  methods 
are  based  on  the  superimposed  coding  technique. 

Sabharwal  and  Bhatia  [4]  proposed  a  method  for  converting  a  2-D  strings  into 
triples.  Each  triple  is  mapped  to  a  unique  hash  address  for  timely  retrieval  of  pictures, 
reducing  the  pattern-matching  problem  corresponding  to  a  query  to  that  of  computation 
of  a  hash  function.  The  values  associated  with  the  picture  objects  are  used  to  compute 
hash  addresses  for  triples  developed  from  the  query.  Heuristics  are  proposed  to  speed 
up  the  computation  of  the  associated  values  for  the  picture  objects.  Experimental  results 
show  that  the  new  algorithm  achieves  almost  a  90%  gain,  in  search  space,  over  existing 
algorithms  to  compute  the  associated  values.  They  also  presented  a  perfect  hash  table 
algorithm  for  image  retrieval  2-D  strings  [42].  In  that  algorithm,  an  additional  heuristic 
leading  to  a  90%  reduction  in  search  space  over  the  earlier  algorithm.  The  new  heuristic 
promises  to  generate  a  minimal  perfect  hash  function  for  each  experimental  data  set, 


7 

which  was  not  possible  with  the  earher  algorithm.  Every  time  the  database  is  modified, 
the  hash  table  needs  to  be  recomputed.  This  limits  the  use  of  the  hash  table  to  applications 
where  the  image  database  is  relatively  fixed.  They  later  presented  an  enhancement  of 
the  perfect  hash  table  to  allow  for  insertion  and  deletion  in  the  hash  table.  [43]  The  new 
hash  table  allows  for  a  relatively  small  number  of  collisions  and  is  called  near-perfect 
hash  table.  It  provides  the  flexibility  of  a  live  database  while  closely  approximating  the 
efficiency  of  retrieval  with  the  perfect  hash  table. 

Chang  and  Lee  [9]  defined  an  ordered  triple  representation  that  transforms  each 
image  or  query  into  a  set  of  ordered  triples  (0„  Oj,  rij),  where  O,  and  Oj  are  two  symbolic 
objects  and  ry  denotes  the  spatial  relationship  between  O,  and  Oj.  They  then  construct  a 
hashing  table  for  all  (0„  Oj,  Ty)  of  all  images  in  the  image  database.  By  searching  the 
preconstructed  hashing  table  for  all  of  the  (O,,  Oj,  rij)  associated  with  a  query,  the  image 
satisfying  that  query  is  easily  determined.  Wu  and  Chang  [52]  applied  the  geometric 
hashing  scheme  to  retrieve  the  most  similar  picture  from  an  iconic  image  database.  They 
transform  each  iconic  picture  or  query  into  a  set  of  ordered  triples  (0„  Oj,  RyYs,  where 
Oi  and  Oj  are  two  iconic  objects  and  is  the  spatial  relationship  code  between  O,  and 
Oj.  They  then  use  a  geometric  hashing  scheme  to  construct  a  set  of  hashing  functions 
corresponding  to  predefined  relationship  codes.  Associated  with  these  hashing  functions, 
the  most  similar  iconic  pictures  satisfying  a  specified  query  can  be  determined. 

A  modified  image  data  structure  based  upon  the  2-D  string,  called  the  modified 
2D  string,  was  introduced  by  Chen  and  Chang  [13].  A  similarity  retrieval  algorithm  for 
the  modified  2D  string,  also  derived  from  the  2-D  string  LCS  algorithm,  proves  more 
efficient  than  the  previous  algorithms. 

A  2D  C-string  for  spatial  knowledge  representation  that  employs  a  cutting  mech- 
anism and  a  set  of  spatial  operators  was  proposed  by  Lee  and  Hsu  [33].  They  extended 


8 

2-D  strings  to  represent  more  types  of  spatial  relationsiiips  between  image  objects  and 
to  provide  for  more  efficient  representation  and  manipulation  of  images.  Since  a  spatial 
relationship  is  a  fuzzy  concept,  the  capability  of  similarity  retrieval  is  essential  in  image 
database  systems.  They  also  suggested  a  spatial  query  and  spatial  reasoning  based  on 
a  2D  C-string  representation  [34].  Huang  and  Jean  [24]  proposed  a  2D  -string  rep- 
resentation scheme  which  extended  the  work  of  Lee  and  Hsu  [33]  of  2D  C-string  by 
including  relative  metric  information  about  the  picture  into  the  strings.  Such  information 
can  be  easily  obtained  from  a  preprocessor  by  counting  the  pixels  of  an  image.  The 
2D  C^-strings  offer  the  following  major  advantages:  (1)  more  accuracy  in  picture  re- 
construction; (2)  less  ambiguity  in  similarity  retrieval;  (3)  reasoning  about  relative  sizes, 
locations,  and  distances  for  a  symbolic  picture  is  possible. 

A  rotation-invariant  spatial  knowledge  representation  called  RS-string  was  proposed 
by  Huang  and  Jean  [25].  The  RS-string  has  the  same  representation  structure  as  the  2D 
C-string.  However,  the  semantics  and  conditions  of  the  spatial  operators  in  RS-strings 
are  defined  by  a  polar  coordinates  system  instead  of  a  Cartesian  coordinates  system.  The 
RS-string  representation  preserves  the  desired  rotation-invariant  property  because  it  is 
based  on  a  polar  coordinates  system. 

The  method  of  relative  coordinates  oriented  symbolic  (RCOS)  strings  [8]  uses  an 
object-oriented  structure  that  avoids  partitioning.  This  method  applies  the  concept  of 
rank  in  simplifying  data  representation. 

Definition  I.  The  rank  of  each  object  symbol  in  a  1-D  string  is  defined  as  the  position 
of  the  object  symbol  minus  the  number  of  symbol  '='  proceeding  this  object  symbol 
in  the  string. 


Let  Xu  and     denote  the  beginning  rank  and  the  end  rank  along  the  x-axis,  respectively, 


and  let  yu  and  denote  the  beginning  rank  and  the  end  rank  along  the  y-axis,  respectively. 
The  tuples  {xn,  yu)  and  fe,  yiz)  stand  for  coordinates  corresponding  to  the  object  symbol 
Oi,  where  (jc,/,  yu)  denotes  the  lower-left  coordinate  and  fe,  y,^)  denotes  the  upper- 
right  coordinate.  The  symbolic  object  O,  is  enclosed  by  a  minimum  bounding  rectangle. 
Taking  Figure  2  as  an  example,  the  corresponding  RCOS  string  is  expressed  as  an  ordered 


3  4 


Figure  2.  A  symbolic  picture  g  for  RCOS  string. 


list  [ABC,  UJ)(2,2),  {2,2){4,4\  (,3,3){5,5)],  where  A,  B,  and  C  are  three  object  symbols 
in  the  picture  g  with  (7,7)  and  (2,2)  denoting  a  pair  of  relative  coordinates  corresponding 
to  A,  with  (2,2)  and  {4,4)  denoting  a  pair  of  relative  coordinates  corresponding  to  B,  and 
with  {3,3)  and  (5,5)  denoting  a  pair  of  relative  coordinates  corresponding  to  C. 

2.2  Related  String  Matching  Methods 

Finding  a  longest  common  subsequence  (abbreviated  LCS)  is  mainly  used  to 
measure  the  discrepancies  between  two  strings.  The  LCS  problem  is,  given  two  strings  X 
and  Y,  to  find  a  maximum  length  common  subsequence  of  X  and  Y.  A  subsequence  of  a 
given  string  is  just  the  given  string  with  some  symbols  (possibly  none)  left  out.  String  Z 
is  a  common  subsequence  of  X  and  F  if  Z  is  a  subsequence  of  both  X  and  Y.  Hirschberg 
[21]  suggested  two  algorithms  to  solve  the  problem.  The  first  algorithm  is  applicable 
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in  the  general  case  and  requires  0{pn+nlogn)  time  where  p  is  the  length  of  the  longest 
common  subsequence.  The  second  algorithm  requires  time  bounded  by  0{p{m+l-p)logn). 
In  the  common  special  case  where  p  is  close  to  m,  this  algorithm  takes  much  less  time 
than  n^.  The  LCS  problem  is  a  special  case  of  the  problem  of  computing  edit  distances. 
The  edit  distance  between  two  character  strings  can  be  defined  as  the  minimum  cost  of 
a  sequence  of  editing  operations  which  transforms  one  string  into  the  other.  Masek  and 
Paterson  [39]  described  an  algorithm  for  computing  the  edit  distance  between  two  strings 
of  length  n  and  m,  n>  m,  which  requires  0(nxmsix{l ,m/logn))  steps  whenever  the  costs 
of  edit  operations  are  integral  multiples  of  a  single  positive  real  number  and  the  alphabet 
for  the  strings  is  finite. 

The  string-to-string  correction  problem,  first  suggested  by  Wagner  and  Fischer 
[49],  determines  the  distance  between  two  strings  as  measured  by  the  minimum  cost 
sequence  of  edit  operations  required  to  change  the  first  string  into  the  other.  The  edit 
operations  investigated  allow  insertion,  deletion  and  change.  This  problem  applies  to 
keypunched  words  for  undoing  certain  common  keypunch  errors.  Lowrance  and  Wagner 
[38]  proposed  the  extended  string-to-string  problem  to  include  in  the  set  of  allowable 
edit  operations  the  operation  of  interchanging  the  positions  of  two  adjacent  symbols. 
This  swap  operation  models  the  error-introduction  process,  such  as  in  typing,  so  that  the 
search  for  the  minimum-cost  edit  sequence  may  identify  errors  better.  An  instance  of  this 
problem,  allowing  only  deletion  and  swap  operations,  was  proven  to  be  an  A^P-complete 
problem  by  Wagner  in  1975. [48] 

The  string  matching  problem,  given  strings  P  and  X,  examines  the  text  X  for  an 
occurrence  of  the  pattern  P  as  a  substring,  namely,  whether  the  text  X  can  be  written  as 
X  =  YPY',  where  Y  and  Y'  are  strings.  String  matching  is  an  important  component  of 
many  problems,  including  text  editing,  bibliographic  retrieval,  and  symbol  manipulation. 
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Several  algorithms  [29,  31,  5]  for  this  problem  have  appeared  in  the  literature.  In  some 
instances,  however,  the  pattern  and/or  the  text  are  not  exact.  For  example,  the  name 
may  be  misspelled  in  the  text.  The  approximate  string  matching  problem  reveals  all 
substrings  in  X  that  are  close  to  P  under  some  measure  of  closeness.  The  most  common 
measure  of  closeness  is  known  as  the  edit  distance,  which  determines  whether  X  contains 
a  substring  P'  that  resembles  P  in  at  most  a  certain  edit  distance  from  P  to  P'.  The  editing 
operation,  for  example,  may  change  one  symbol  of  a  string  into  another,  delete  a  symbol 
from  a  string,  or  insert  a  symbol  into  a  string.  Ukkonen  [47]  constructed  a  deterministic 
finite-state  automaton  that  solves  the  problem.  Some  other  approximate  string-matching 
algorithms  can  be  found  in  the  related  literature  [51,  1,  2]. 

2.3  Related  Image  Database  Systems 

IIDS:  IIDS  (Intelligent  Image  Database  System)  [11]  is  based  on  an  image  data 
structure,  2-D  strings.  2-D  strings  provide  efficient  means  for  iconic  indexing  in  image 
database  systems,  and  spatial  reasoning.  A  picture  query  can  also  be  specified  as  a 
2-D  string.  The  problem  of  pictorial  information  retrieval  then  becomes  one  of  2-D 
string  subsequence  matching.  IIDS  supports  spatial  reasoning,  flexible  image  information 
retrieval,  visualization,  and  traditional  image  database  operations. 

QBIC:  The  QBIC  (query  by  image  content)  system  allows  users  to  find  pictorial 
information  in  large  image  and  video  databases  based  on  color,  shape,  texture  and 
sketches.[17]  Therefore,  QBIC  techniques  serve  as  a  database  filter  and  reduce  the 
search  complexity  for  the  user.  These  techniques  limit  the  content-based  features  to 
those  parameters  that  can  be  easily  extracted,  such  as  color  distribution,  texture,  shape 
of  a  region  or  an  object,  and  layout.  The  system  offers  a  user  a  virtually  unlimited  set  of 
unanticipated  queries,  thus  allowing  for  general  purpose  applications  rather  than  catering 
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to  a  particular  application.  Color-  and  texture-based  queries  are  allowed  for  both  images 
and  objects,  whereas  shape-based  queries  are  allowed  only  for  individual  objects  and 
layout-based  queries  are  allowed  only  for  an  entire  image. 

Photobook:  Photobook  is  a  set  of  interactive  tools  for  browsing  and  searching  an 
image  database.  [41]  The  features  used  for  querying  can  be  based  on  both  text  annotations 
and  image  content.  The  key  idea  behind  the  system  is  semantics-preserving  image 
compression,  which  reduces  images  to  a  small  set  of  perceptually  significant  coefficients. 
These  features  describe  the  shape  and  texture  of  the  images  in  the  database.  Photobook 
uses  multiple  image  features  for  querying  general  purpose  image  databases.  The  user  is 
given  the  choice  to  select  features  based  on  the  appearance,  shape,  and  texture  to  browse 
through  large  databases.  These  features  can  be  used  in  any  combination  and  with  textual 
features  to  improve  the  efficiency  and  accuracy  of  the  retrievals. 

Jain  and  Vailaya  [26]  proposed  a  method  for  trademark  image  database  retrieval 
based  on  object  shape  information  that  would  supplement  traditional  text-based  retrieval 
systems.  This  system  uses  a  two-stage  hierarchical  retrieval  system:  (i)  a  simple 
statistical  feature-based  process  quickly  browses  through  a  database  for  a  moderate 
number  of  plausible  retrievals;  and  (ii)  a  deformable  template  matching  process  screens 
the  candidate  set  for  the  best  matches.  The  proposed  hierarchical  content-based  image 
retrieval  algorithm  has  been  tested  on  a  trademark  image  database  containing  1100  images. 
The  retrieval  system  is  insensitive  to  variations  in  scale,  rotation,  and  translation.  In  other 
words,  even  if  a  query  image  differs  from  its  stored  representation  in  the  database  in  its 
orientation,  position,  or  size,  the  image  retrieval  system  is  able  to  correctly  retrieval  it. 


CHAPTER  3 
A  LINE-STRING  IMAGE  DATA  STRUCTURE 


String  data  structures  predominate  in  image  databases  [8,  9,  10,  33,  36];  neverthe- 
less, they  lack  several  requirements  of  databases.  The  following  weak  points  are  typical 
in  string  data  structures: 

1.  String  image  data  structures  for  symbolic  objects  portray  the  spatial  relations  of 
centroids  or  extreme  points  of  objects,  rather  than  the  actual  relative  positions 
of  objects. 

2.  Symbolic  object  representation  falls  short  of  characterizing  an  object's  structural 
shape  and  size.  Although  RCOS  strings  represent  each  object  by  its  beginning 
boundary  and  end  boundary,  they  still  lack  the  object's  overall  configuration. 

3.  Some  objects  can  be  detected  but  cannot  be  recognized.  In  these  cases,  im- 
age retrieval  depends  on  the  objects'  configurations,  a  requirement  generally 
unachievable  with  symbolic  objects. 

Instead  of  using  point  symbols  for  object  representation,  line  segments  represent  an 
object's  skeleton.  This  representation  captures  the  basic  shape,  size,  and  relative  position 
of  an  object;  the  data  structure  based  on  this  representation  avoids  the  above  mentioned 
problems. 

Figure  3  outlines  our  proposed  method.  We  first  skeletonize  the  segmented  image. 
A  linear  approximation  algorithm  converts  the  image  skeleton  into  line  segments.  The 
endpoints,  relative  lengths,  and  relative  angles  of  these  straight  lines  are  then  stored  in 
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a  line-string  data  structure. 
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Figure  3.  System  structure  and  transformations  in  image  representations. 


3.1  A  Linear  Image 


Zhang-Suen's  method  [53]  is  used  to  skeletonize  a  segmented  image.  Figure  4 
shows  an  image  and  its  skeleton.  Next  the  linear  approximation  algorithm  converts  the 
skeleton  into  line  segments.  The  approximation  captures  the  essence  of  a  skeleton  in  the 
fewest  possible  line  segments.  A  polygonal  approximation,  based  on  an  error  function 
[40],  is  applied  to  this  method.  Before  applying  this  algorithm,  however,  the  skeleton 
has  to  be  smoothed  to  unit  thickness  so  that  branch  skeletons  may  be  located.  Let  e  be 
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the  maximum  allowable  error.  For  a  given  point  A,  through  which  an  approximating  line 
must  pass,  one  can  define  two  points  B  and  C  at  a  distance  e  from  A.  The  algorithm 
searches  for  the  longest  segment  where  the  curve  is  contained  between  two  parallel 
tangents  starting  from  B  and  C. 

APPROX_LINE  (IMG,  p„  pj,  e) 


/*  IMG:  a  skeletal  image  */ 
/*  Pi.'  the  initial  point  of  an  object's  skeleton  */ 
/*  pj:  the  neighbor  of  pi  at  the  skeleton  */ 
/*  e:  the  maximum  allowable  error  */ 

1  b  <—  Pi  +  e 

2  c      Pi  -  e 

3  IS_FIRST  ^  TRUE 

4  while  NUMBER_8_NEIGHB0R(pi)  /  0 

5  while  NUMBER_8_NEIGHB0R(Pi)  >  1 

6  Pj  ^  BEST_ROUTE(IMG,  Pi,  e) 

7  APPROX_LINE(IMG,  Pi,  pj,  e) 

8  print  Pi 

9  IMGlpi]  ^  CLEAR 

10  Lb  ^bpi 

11  Lc  <—  cpi 

12  if  IS_FIRST 

13  Labove  < —  Lb 

14  Lbelow  ^  Lc 

15  IS_FIRST  ^  FALSE 

16  else 
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17  if  Lb  is  above  habove 

18  Labove  < —  Lb 

19  if  Lc  is  below  Lbeiow 

20  Lbeiow  < —  Lc 

21  if  I  [Labove,  Lbeiow)   >  0 

22  print  p, 

23  b  <—  Pi  +  e 

24  c      Pi  -  e 

25  IS_FIRST  ^  TRUE 

26  Pi  ^  pj 

27  Pj  ^  8_NEIGHB0R(pi) 

28  IMG[pi]  ^  CLEAR 

29  print  Pi 

The  function  BEST_ROUTE  finds  the  route  of  the  longest  line  segment  when 
Pi  has  more  than  one  8-neighbor.  This  is  why  the  skeleton  needs  smoothing  before 
applying  the  algorithm.  A  non-unit  thickness  skeleton  may  mislead  the  algorithm  into 
calling  the  BEST_ROUTE  function.  This  algorithm  requires  quadratic  time  because  the 
BEST_ROUTE  function  needs  to  check  as  many  routes  as  possible  and  any  point  on  the 
skeleton  may  invoke  it.  Figure  4  (c)  shows  the  linear  image  that  is  the  output  of  this 
algorithm  whose  input  is  the  skeletal  image  of  Figure  4  (b). 
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(a)  (b)  (c) 


Figure  4.  (a)  The  original  image,  (b)  the  skeletal  image,  and  (c)  the  linear  image. 

3.2  Representation  and  Examples 

After  an  image  has  been  transformed  to  a  linear  image,  an  IDBS  stores  the  image 
in  its  relational  database  through  a  data  structure,  called  a  line  string,  that  maintains  line 
segments  in  a  string.  Lines  or  points  are  described  in  polar  coordinate  systems.  Each 
line  or  point  in  the  plane  is  assigned  coordinates  (r,  6),  where  r  is  the  length  and  6  is  the 
measure  of  an  angle  with  an  initial  side  corresponding  to  the  polar  axis  and  a  terminal 
side  coinciding  with  the  line  or  point. 

To  express  the  representations  of  line  strings,  let  L,  P,  and  R  be  three  sets  of 
symbols.  Each  symbol  from  L  represents  a  line  segment;  each  symbol  from  P  represents 
a  point  or  a  centroid;  and  R  =  { 'e',  '=' },  where  e  is  an  empty  string.  A  line  string  written 
as  {rc,0c,rp,0p,ri,ei),  r  sorted  by  length  and  6  sorted  by  angle,  is  defined  as 

(Ci  7  C2  7   ...  7  C„,  C;(i)  7  C^(2)  7   ■  •  •  7 
Pll  P2f  ■■■  7  Pn',  Pg(l)  7  Pg{2)  7  •  •  •  7  Pg{n'), 
hlhl  ■■■  1  In",  lh(\)  7  k{2)  7  •  •  •  7  h(n")), 

where 

7  is  over  R,  and  7  is  '='  if  the  ranks  of  the  symbols  on  both  sides  are  equal,  otherwise 
7  is  e  with  rank  being  defined  as  in  Chapter  2; 
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C/  7  ^2  7  •  •  •  7  c„  is  a  string  of  centroids  over  P  with  every  two  centroids  separated  by 

a  7  and  with  a  centroid  c  as  the  center  of  each  object's  minimum  enclosed  rectangle; 

Pi  1  Pi  1  •  •  •  1  Pn-     ^  String  of  points  over  P  with  every  two  points  separated  by  a 

7  and  with  a  point  p  as  an  endpoint  of  a  line  segment  or  an  object  symbol; 

//  7  /2  7  •  •  •  7  4  '  is  a  string  of  line  segments  over  L  with  every  two  segments 

separated  by  a  7  and  with  a  line  segment  /  as  an  object's  skeleton  line;  and 

f,  g,  and  h  are  permutation  functions  on  positive  integers. 


(a)  (b)  (c) 

Figure  5.  (a)  The  original  image,  (b)  the  image  with  centroids  marked,  and  (c) 
centroids  in  the  polar  coordinate  system.  The  number  in  parentheses  is  a  length 
rank,  and  the  angle  ranks  can  be  derived  easily  from  the  picture  by  positions. 


Figure  6.  (a)  A  linear  image,  (b)  endpoints  in  the  polar  coordinate 
system,  and  (c)  linear  skeleton  in  the  polar  coordinate  system. 
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For  example,  the  centroids  in  Figure  5  can  be  represented  by  a  string  of  centroids: 

(re,  e,)  =  (4  1  7,  1  7  4).  (2) 

The  endpoints  in  Figure  6  can  be  represented  by  a  string  of  endpoints  (rp,0p).  The 
elaborate  form  of  line  string,  only  for  clarification  and  not  to  be  read  by  computers,  is 
shown  first: 

elaborate  form  of  (r^,  Oj,) 

(3) 

=  i4b  4d  4c  la  h  4a  7c  7b  7a,  h  7c  h  4b  4d  7a  4a  7j  4c), 

and  the  corresponding  line  string  is 

(r,,  Op)  =  {4  4  4  1  1  4  7  7  7,  1  7  1  4  7  4  7  4  4).  (4) 

The  linear  skeleton  in  Figure  6  can  be  represented  by  a  string  of  skeleton  lines(r-i,^i): 

elaborate  form  of  (r/,  Oi) 

(5) 

=  {4"  1'  7"  4"'  4'  7\  7"  4"'  4'  7'  1'4"), 
and  the  corresponding  line  string  is 

iri,0,)  =  (4  1  7  4  4  7,  7  4  4  7  1  4)-  (6) 

Combining  the  three  line  strings  (2),  (3),  and  (5)  results  in  the  line  string's  elaborate 

form  for  (a)  of  Figure  4: 

elaborate  form  of  {rc,6c,rj,,9p,ri,6i) 


=  {4  1  7,  1  7  4, 

4b  4d  4c  la  h  4a  7c  7i  7^,  h  7c  la  4b  4d  7a  4a  7*  4c, 

4"  1'  7"  4"'  4'  7',  7"  4"'  4'  7'  1'  4"), 
and  combining  the  three  line  strings  (2),  (4),  and  (6)  produces  the  line  string 

irc,6c,rp,ep,ri,ei) 

=  (4  1  7,  1  7  4, 

4  4  4  1  1  4  7  7  7,  1  7  1  4  7  4  7  4  4, 

417447,744714). 


(7) 


(8) 


CHAPTER  4 
AN  OBJECT-ORffiNTED  LINE-STRING 
SPATIAL  KNOWLEDGE  REPRESENTATION 

Although  the  line-string  image  data  structure  is  efficient  in  storage  and  concise 
in  representation,  users  may  want  a  more  structural  and  clear  way  of  describing  spatial 
relationships  in  database  queries.  To  this  end,  we  present  an  object-oriented  method 
to  specify  a  line  string.  The  following  provides  a  grammar  for  this  representation  and 
the  presents  examples.  The  algorithms  of  image  reconstruction  and  retrieval  also  are 
discussed. 

4. 1  Representation  and  Examples 

The  spatial  knowledge  representation  of  an  image  in  database  queries  can  be 
expressed  through  an  OOLS.  A  context-free  grammar  for  an  OOLS  is  given  by  the 
following: 


OOLS 

{object  objects} 

object 

[id  centroid  points  lines] 

objects 

object  objects  \  e 

centroid 

(rp,Op)  1  e 

points 

,{Tp,6p)  more-points  \  € 

more-points 

{Tp,Op)  more-points  \  e 

lines 

,(pi,Pj,Ti,9t)  more-lines\e 

more-lines 

(pi,Pj,r/,^i)  more-lines  \  c 
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Using  an  OOLS,  a  database  user  has  an  organized  way  of  forming  an  objects'  spatial 
relationship  in  a  query.  In  this  grammar,  the  fourteen  terminal  symbols  are 


where  id  is  an  object  symbol  and  where  r,,  and  are  the  length  and  angle  ranks, 
respectively,  of  a  centroid  or  point  p.  Line  segment  /  =  (p„  p,,  r;,  Oi)  has  p,  and  py 
as  its  initial  and  end  point  sequence  numbers  and  has  r/  and  6,  as  its  length  and  angle 
ranks.  The  eight  nonterminals  are 

OOLS  object  objects  centroid  points  more-points  lines  more^lines, 

in  which  OOLS  is  the  start  symbol,  centroid  is  for  centroids,  points  is  for  endpoints,  and 
lines  is  for  line  segments.  Every  nonterminal  except  OOLS  and  object  must  either  be 
provided  with  a  full  content  or  with  an  empty  string  e. 

To  clarify  the  semantic  concept  of  OOLS,  let  O,  P,  and  L  be  sets  of  symbols.  Each 
symbol  from  O  represents  an  object  symbol.  Each  symbol  from  P  is  an  endpoint  of  a  line 
segment,  an  object  point  or  the  centroid  of  an  object.  A  symbol  from  L  represents  a  line 
segment.  The  following  OOLS  elliptical  form  is  used  to  explain  its  semantic  implication. 


where 

O,,  .  . .  ,0„  are  n  object  symbols  over  O; 

in,  Oi)  is  the  representation  of  an  object  (9,'s  centroid  over  P,  where  r,  and  6^  are  its 
length  and  angle  ranks  among  all  centroids; 


id  rp   fp   r;   6>/   pi   p^-    ','  {  }    [  ]   (  ), 


{  [Oi  (r, ,  e,),  (rp,^ ,  ^p„) . . .  (rp,, ,  ^p^J,        pp.n,, ,  O^,, )...       p«,r,,,,         •  •  • 
0,{r„d,),  (rp.,,^p„)...  (rp_^,,^p„,),  (/'a',  ,       . . .  (py ,  p,,,r,_^,,  ... 

0„{r„,e„),  (rp„, ,  ^p„, )  .  . .  (rp_^^„,  ^p_^^„) ,  {pa" ,  Pi3"  ,r,^.j ,  0,^, )...  (py',P6",r,  0, 

\  3  "y 


22 

(rp,^,  ^pj  is  the  representation  of  an  object  0,'s  endpoint  over  P,  where  r,,^  and 
^P,^  are  its  length  and  angle  ranks  among  all  endpoints; 

(p^,P5,  r/,^,^;J  is  the  representation  of  an  object  0,'s  line  segment  ly  over  L,  where 
p-y  and  PS  are  its  initial  and  end  point  sequence  numbers  and  where  r/,^  and  0/,^  are  its 
length  and  angle  ranks  among  all  skeleton  lines.  The  lines  always  point  downward 
and  point  to  the  right  for  horizontal  lines. 

Take  the  linear  image  at  Figure  7  (c)  as  an  example.  Its  centroids,  endpoints,  and  skeleton 

lines  in  the  polar  coordinate  systems  are  shown  in  Figure  8  (c).  Figure  9  (b),  and  Figure 

9  (c),  respectively.  The  following  steps  explain  how  to  represent  the  linear  image  by  an 

OOLS.  The  OOLS'  elaborate  form  is  shown  first: 

OOLS'  elaborate  form  of  Figure  7  (c) 

={[A{A,),  («,)(a,)(a5)(«4),  («')(«")(«"')] 
[5(5,),  (6,)(6,)(6,)(6,),  (6')  (6")  (6'")]}, 

and  the  OOLS  is 

OOLS  of  Figure  7  (c) 

={[A(2,2),  {8,7)i6,8){l,6)i3,3),  (1 ,  2,4,  2){2,  3,  5 , 4){3, 4,  S,  6)] 

[B{1,1),  i5,5){7,4){4,2){2,l),  {1,2,2,5){2,3,6,3){3,4,1,1)]}. 
A  retrieval  query  specified  via  an  OOLS  expression  needs  to  be  translated  into  a 
line-string  query  before  it  can  be  satisfied.  An  OOLS  includes  more  information  than  its 
corresponding  line  string,  e.g.,  an  OOLS,  unlike  a  line  string,  specifies  the  two  endpoints 
of  a  line.  Thus,  an  OOLS  could  easily  translate  into  a  line  string.  Translate  an  OOLS 
into  a  line  string  without  adversely  affecting  system  performance:  The  translation  takes 
0(N)  time  for  symbols  with  each  query  transformed  just  once.  Moreover,  the  extra 
space  adds  no  cost  to  system  memory  because  the  IDBS  stores  an  image's  line  string, 
rather  than  its  OOLS. 


(a) 


(b) 


(c) 


Figure  7.  (a)  The  original  image,  (b)  the  skeletal  image,  and  (c)  the  linear  image. 


(a) 


(2)  Ac 


Bed) 


(b) 


(c) 


Figure  8.  (a)  The  original  image,  (b)  the  image  with  centroids 
marked,  and  (c)  centroids  in  the  polar  coordinate  system. 
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Figure  9.  (a)  The  linear  image,  (b)  endpoints  in  the  polar  coordinate 
system,  and  (c)  linear  skeleton  in  the  polar  coordinate  system. 


4.2  Image  Restoration  and  Retrieval 
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An  OOLS  can  reconstruct  a  linear  image  that  as  long  as  the  configurations  of 
objects  are  known  may  restore  a  segmented  image.  The  reconstruction  transform,  which 
deciphers  the  length  and  angle  information  in  an  OOLS  to  approximate  line  segments,  is 
described  below.  The  notation  used  in  this  description  arises  from  the  OOLS  grammar. 

Algorithm  I.  Image  restoration  from  an  OOLS 

Input.  An  OOLS  spatial  knowledge  representation. 

Output.  A  linear  image. 

begin 

1.  Divide  the  first  quadrant  of  a  polar  coordinate  system  into  Y^\points\  Chapters. 
Assign  the  point  p  with  an  /th  angle  rank  to  the  ith  Chapter,  and  set  the  length 
of  p  to  a  length  d  \i  p  has  a  dih  length  rank. 

2.  Connect  the  endpoints  of  each  line  segment  /  according  to  the  first  two  elements 
of  the  lines  iPi,Pj,'''hOi). 

3.  For  each  line  segment  I,  there  may  exist  two  different  ranks  for  r,  or  9,.  One 
rank  is  from  the  line  just  created;  another  is  from  lines  {pi,Pj,ri,0i).  Adjust  / 
then  by  taking  the  average  values  for  r,  and 

4.  Divide  the  first  quadrant  of  a  polar  coordinate  system  into  T,\object\  Chapters. 
Assign  the  center  of  an  object's  skeleton  line  to  the  ith  Chapter  if  the  object's 
centroid  has  an  ith  angle  rank,  and  move  the  center  from  the  origin  by  d  if  the 
centroid  has  a  dth  length  rank. 

end 
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Figure  10  shows  the  reconstructed  Unear  image  of  Figure  6  (a).  The  dotted  Hues  show 
the  division  of  a  quadrant  into  sections. 


Figure  10.  (a)  The  Hnear  image  from  (a)  of  Figure  6,  (b)  the  reconstructed  image  after 
the  first  two  steps  of  the  restoration  algorithm,  and  (c)  the  final  reconstructed  image. 


Line  strings  can  characterize  various  kinds  of  spatial  relationships  described  in 
other  string  image  representations,  but  can  they  retrieve  images  correctly  and  effectively? 
The  spatial  relationship  in  an  image  query  can  be  specified  through  any  one  of  the 
following  five  methods:  a  line  string,  an  OOLS,  a  linear  image,  a  skeletal  image,  or 
a  segmented  image.  The  last  four  representations,  however,  need  conversion  to  line 
strings  before  processing.  Image  retrieval  then  becomes  line-string  matching,  i.e.,  finding 
a  longest  approximate  common  subsequence  [21]  between  two  images.  Therefore,  an 
image  retrieval  algorithm  is  needed  to  determine  a  maximum-gain  approximate  common 
subsequence  of  two  sequences,  which  will  be  discussed  later. 


CHAPTER  5 
ANALYSIS  OF  LINE  STRINGS 


Compared  with  other  image  data  structures,  line  strings  concisely  describe  elaborate 
spatial  relationships  of  objects.  The  data  structure  requires  relatively  few  symbols  to 
create  numerous  line  connections  and  distinct  spatial  relations.  This  Chapter  analyzes 
the  spatial  relationships  and  space  needs  of  line  strings  and  provides  comparisons  of 
storage  space  and  computational  complexity  between  the  method  discussed  and  other 
string  image  data  structures. 

5.1  Number  of  Spatial  Relations 

Theorem  V  gives  the  number  of  spatial  relations  of  a  line  string.  It  is  constructed 
through  a  sequence  of  lemmas  and  theorems  that  establish  the  numbers  of  spatial  relations 
of  line  string  {rc,0c,rp,Op,ri,di)  components;  i.e.,  r,,  6„  rp.  Op,  r,,  and  9,.  To  establish 
these  lemmas  and  theorems,  we  need  to  define  the  notion  of  multisets,  which  are  sets 
allowed  to  contain  repetitive  elements. 

Definition  IL  A  multiset  is  like  a  set  except  that  it  can  have  repetitions  of  identical 
elements.  If  S  and  T  are  multisets,  we  define  new  multisets  S  \i)T,  S  UT,  and  S  nT 
in  the  following  manner:  An  element  occurring  exactly  n  times  in  S  and  m  times  in 
T  occurs  exactly  n+m  times  in  5  i±)  T,  exactly  max(a,  b)  times  in  SUT,  and  exactly 
min(a,  b)  times  in  5  n  T.  [30] 
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For  example,  consider  the  endpoint  multiset  illustrated  by  Figure  9.  This  multiset  is 
given  by 

T  =  {a,  a,  a,  a,  b,  b,  b,  6} 

and  contains  four  a's  and  four  b's.  The  nth  power  of  a  multiset  T  is  defined  next. 

Definition  III.    If  T  is  a  multiset  and  n  is  a  nonnegative  integer,  then 

y„  ^  f  0  if  n  =  0 

It"-'  wr  otherwise 

Our  first  lemma  establishes  the  number  of  spatial  relations  of  the  centroids,  i.e., 
or  9c  in  a  line  string.  In  the  following,  let  NC  and  NE  denote  the  number  of  centroids 
and  the  number  of  equal  signs  between  these  NC  centroids,  respectively. 

Lemma  I.  For  the  r<-  or  9^  in  a  line  string  {rc,dc,rp,6p,ri,0i)  with  given  numbers 
of  centroids  NC>0  and  with  equal  signs  between  these  centroids  0<NE<NC-1,  the 
number  of  spatial  relations  NSR[.  of     or  9^  is  given  by 

NSR:iNC,NE)={Jf^^]{NC  -  NE)\ 

where  "{  }"  is  the  notation  for  the  Stirling  number  of  the  second  kind[30]. 

Ascertained  from  the  above  lemma,  the  number  of  spatial  relations  of  {r^,  9 J  is  the 
product  of  NSR'^{NC,NEr)  and  NSR'^{NC,NEg). 

Theorem  I.  For  the  and  9,  in  a  line  string  (r^,  9c,rp,  9p,ri,ei)  with  given  numbers 
of  centroids  NC>0,  equal  signs  NE,  between  the  centroids  in  r,  with  0<NE,<NC-1, 
and  equal  signs  NEq  between  the  centroids  in  9^  with  0<NEq<NC-1,  the  number  of 
spatial  relations  NSR^  of  {r^ft^)  is  given  by 

NSR,{NC,  NE,,  NEe)=NSB:^{NC,  NE,)NSR[{NC,  NEe). 


28 


(Cl=C2  C3=C4) 

(C3=C4  Cl=C2) 

(Cl=C3  C2=C4) 

(C2=C4  Cl=C3) 
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Figure  11.  Fourteen  spatial  relations  of  9^  in  a  line 
string  with  four  centroids  and  two  equal  signs. 


Consider    Figure     11     as    an    example.  In    this    example,     we  have 

iV5i?^(4,2)  =  {^}2!  =  7  X  2!  =  14  spatial  relations  for  9,  in  a  line  string  with 
NC=4  and  NE=2. 

Lemma  II  provides  an  equation  for  finding  the  number  of  spatial  relations  of 
or  9,,  in  a  line  string.  The  collection  of  nonempty  endpoint  multisets  5,,={object  h's 
endpoints},  l<h<NO  is  a  partition  of  the  multiset  of  all  endpoints  U.  The  collection  of 
nonempty  multisets  «,xr,={ endpoints  separated  by  '='s},  l<i<I  with  «,>0  is  a  partition 

NO 

of  multiset  j±)^  Sh-  Each  multiset  T,  is  either  a  single  endpoint  multiset  or  an  '='  separates 
every  two  endpoints  in  that  multiset. 
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Lemma  II.  For  the  r,,  or  Op  in  a  line  string  {rc,0c,rp,6p,ri,9i)  with  given  num- 
bers of  objects  NO>0,  endpoints  NP>NO,  and  equal  signs  between  these  endpoints 
0<NE<NP-],  the  number  of  spatial  relations  NSR'p  of     or  9p  is  given  by 

NSR'^{N0,NP,NE) 

E 


NSRp{NO,NP,NE),  and 

f 

NSRl{NO,NP,NE) 


NO  .  ■  ^  ^^p, 


J2       {NP  -  NE)\ 

I  NO  I 

.^:^:'  =  .^:^'.        n  n,! 


where  E  n,=  NP  -  NE. 


Theorem  II,  which  is  a  consequence  of  Lemma  II,  provides  the  number  of  spatial  relations 
of  {rp,9p).  The  number  of  spatial  relations  of  {rpfip)  is  equal  to 
NSRpiNO,  NP,  NE,,  NEe) 

^  ^  [NSRp{NO,NP,NEr)NSR'piNO,NP,NEe) 

-ILL'p{NO,NP,NEr,NEe)]. 

Among  E         NSR'p{NO,NP,NEr)NSR''{NO,NP,NEg),  the  number  of  ille- 

5*=t/,|C/|=NP 

gal  spatial  relations  is 

ILLp{NO,NP,NE,,NEe) 

=  NO    ^  ILL'p{NO,NP,NE,,NEe). 

h=l  Sh  =  U,\U\  =  NP 

ILLp{NO,NP,NEr,NEg)  >  0  when  the  statement  (\ShnTi\  >  \\Sh\/2])  A 
{\ShnRj\  >  \\Sh\/2]),  which  means  at  least  two  endpoints  of  object  h  are  superimposing 
on  each  other,  is  true.  The  nonempty  endpoint  multisets  5;,= {object  /I's  endpoints}, 
l<h<NO  are  from  Lemma  11.  The  collection  of  nonempty  multisets  n,x 7,= {endpoints 
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in  fp  separated  by  '='s},  l<i<I  with  n,>0  and  the  collection  of  nonempty  multisets 

NO 

m,x/?y={endpoints  in  6p  separated  by  '='s},  l<i<J  with  mj>0  are  two  partitions  of  1+1  Sh- 

Theorem  II.  For  the  r,,  and  Op  in  a  line  string  (r^,  ^c,'*p,  ^p,  ^/)  with  given  numbers 
of  objects  NO>0,  endpoints  NP>NO,  equal  signs  NEr  between  the  endpoints  in  r,,  with 
0<NEr<NP-l,  and  equal  signs  NEq  between  the  endpoints  in  6p  with  0<NEq<NP-1, 
the  number  of  spatial  relations  NSRp  of  {rp,6p)  is  given  by 


NSRp{NO,NP,  NEr.NEg) 


^  NSRp{NO,NP,NEr,NEg), 

W  Sh=U,\U\=NP 


NO 

)i=i 


NSRp  {NO,  NP,  NEr,  NEg) 

=  NSRp{NO,NP,  NEr)NSRl{NO,NP,  NEg)- 

ILL'p{NO,NP,  NEr,  NEg),  and 

ILL'p{NO,NP,NEr,NEg) 
0 


(NP-NErY 

I  NO  III 

.w^:'=w,5.  11".! 


}2  {NP-NE^y. 

J  NO  J 


2/  P  is  false 
otherwise 


where  statement  P  =  (|5,,nr,|  >  [|5,^|/2l)A(|5',,ni?,|  >  r|5;,|/2]),  E  n,=iVP  -  NEr, 
and  E  m,  =  ^■p  -  7V£^. 

i=i 


Forexample,the/iXp(2,3,l,l)=  E  (  E  ^fl,)  E  iftl)  =  E  (ifn)  (iTTt)  =  4  +  4  =  8  il- 
legal spatial  relations  of  a  line  string  with  N0=2,  NP=3,  NE,=1,  and  NEq=1  are  shown 
in  Figure  12. 
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(Oi=Oi  02,  Oi=Oi  02)     {02  0i=0i,  Oi=Oi  O2)     (Oi=Oi  O2,  O2  Oi=Oi)      (02  0i=0i,  02  0i=0i) 
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Figure  12.  Eight  illegal  spatial  relations  of  a  line  string  with  two 
objects,  three  endpoints,  one  equal  sign  in  r^,  and  one  equal  sign  in  r^. 

Theorem  III  calculates  the  number  of  spatial  relations  of  (r,,^,).  Lemma  III,  which 
is  for  constructing  Theorem  III,  provides  a  formula  to  find  the  number  of  spatial  relations 
of  n  or  Ol  in  a  line  string.  The  nonempty  endpoint  multisets  5A={object  /I's  endpoints}, 
l<h<NO  are  from  Lemma  11.  The  collection  of  line  segment  multisets  r,={object  /'s 
line  segments},  l<i<NO  with  0<ir,l<^'"^"'^  is  a  partition  (allowed  empty  multisets)  of 
the  multiset  of  all  line  segments  V.  The  collection  of  nonempty  multisets  myX/?,={line 

NO 

segments  separated  by  '='s},  l<j<J  with  mj>0  is  a  partition  of  multisets  l+l  T,. 

1=1 

Lemma  IIL  For  the  r,  or  ^,  in  a  line  string  (rc,dc,rpj0p,ri,ei)  with  given  numbers 
of  objects  NO>0,  endpoints  NP>NO,  line  segments  0<NL<(^^^-^"+^y  and  equal 
signs  between  the  line  segments  0<NE<NL-1,  the  number  of  spatial  relations  NSB!^ 

of  ri  or  di  is  given  by 

NSB!j{NO,NP,NL,NE) 

E 


E 


MO     -  NSR,{NO,NL,NE),and 


NSR,  {NO,NL,NE) 

J2        {NL  -  NEy. 

~      J  NO  J 
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where  E  m,=  NL  -  NE. 


Theorem  III,  which  is  a  consequence  of  Lemma  III,  provides  the  number  of  spatial 
relations  of  (r,,^,).  The  number  of  spatial  relations  of  (r;,^/)  is 

NSR,{NO,  NP,  NL,  NE^,  NEe) 

=  NO   ^         NO  ^  [nsr';{no,nl,ne.) 

W  Sh-U.lUl-NP       .W  T,=V,\V\=NL 
n.=  l  t  =  l 

NSR'iiNO,  NL,  NEe)  -ILL\{N0,  NL,  NE,,NEe)]. 


Among  ^  NO     ^        N S R'l {NO,  N L,  N Er)N SR'i {NO,  NL,N Eg),  the 

Sh=U,\U\==NP     _W  T,  =  V,\V\=NL 

number  of  illegal  spatial  relations  is 


ILL,{NO,  NP,  NL,  NE,,  NEe) 

=  No    ^  NO    ^  ILL\{NO,NL,NE,,NEe). 

W  S^=U,\U\=NP        W  Ti  =  V,\V\=NL 


ILLi{NO,NP,NL,NEr,NEg)  >  0  when  the  statement  {\TinRj\  >  \\Ti\/2])  A 
{{TiHQkl  >  [|ri|/2]),  which  means  at  least  two  Hne  segments  of  object  /  are  super- 
imposing on  each  other,  is  true.  The  nonempty  endpoint  multisets  5/,  and  line  segment 
multisets  T,  are  from  Lemma  EI.  The  collection  of  nonempty  multisets  m,xi?y={line 
segments  in  r,  separated  by  '='s},  l<j<J  with  mj>0  and  the  collection  of  nonempty 
multisets  OkXQk={line  segments  in  9i  separated  by  '='s},  l<k<K  with  Ok>0  are  two 

NO 

partitions  of  l+J  Ti. 

t=i 

Theorem  III.  For  the  r,  and  0,  in  a  line  string  (r^,  ec,rp,  9p,  ri,  Oi)  with  given  numbers 
of  objects  NO>0,  endpoints  NP>NO,  line  segments  0<A^L<(^^-^^+^),  equal  signs 
NEr  between  the  lines  in  r,  with  0<NEr<NL-l,  and  equal  signs  NEq  between  the  lines 
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in  9,  with  0<NEq<NL-1,  the  number  of  spatial  relations  NSRi  of  ir,,6,)  is  given  by 


NSR,{NO,  NP,  NL,  NE,,NEe) 

=  ^o  ^        NO  ^  nsr';'{no,nl,ne^,ne,), 

y  Sh=U,\U\=NP        W  T,  =  V,\V\=NL 

n—1  t=l 

NSR'i'iNO,  NL,  NE, ,  NEg) 

=  NSR"{N0,  NL,  NEr)NSR'i{NO,  NL,  NEe)- 
ILL',{NO,NL,NE,,NEe),  and 


ILL,{NO,NL,NEr,NEe) 
0 


NO  J 


J2  (NL-NEg)'. 


K  NO 


Ok'- 


if  P  is  false 


otherwise 


where  statement  P  =  (|T,ni?j|  >  [|ri|/2])  A  (\TinQk\  >  [1^^1/2]),  E  mj=NL  -  NEr, 

K 

and  E  Ofc=  iVX  -  NEe. 


Lemma  II  and  Theorem  II  are  to  find  the  number  of  spatial  relations  of  (rp,9p),  so 
are  Lemma  III  and  Theorem  IE  to  (r,,^,)'s.  To  find  the  number  of  spatial  relations  NSRp, 
of  irp,Op,ri,d,),  we  only  need  to  integrate  these  lemmas  and  theorems  into  Theorem  IV. 

Theorem  IV.  For  the  r,„  Op,  r,  and  9,  in  a  line  string  irc,Oc,rp,ep,ri,di)  with  given 
numbers  of  objects  NO>0,  endpoints  NP>NO,  line  segments  0<NL<(^^^~^°+^y 
equal  signs  NEr^  between  the  endpoints  in  with  0<NEr^<NP-l,  equal  signs  NEq^ 
between  the  endpoints  in  9p  with  0<NEe^<NP-l,  equal  signs  NEr,  between  the 
lines  in  r,  with  0<NEr,<NL-l,  and  equal  signs  NEg,  between  the  lines  in  9,  with 
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0<NEg,<NL-l,  the  number  of  spatial  relations  NSRpi  of  {rp,6p,rift,)  is  given  by 
NSRp,{NO,  NP,  NL,  NE,^,  NE^^,  NE^^.NE,,) 

^        NSRl'  {NO,  NP,  NE,^ ,  NE^^) 


NO  '  ■  p 

W  Sh=U,\U\  =  NP 

NO 

.W  T,  =  V,|V|=WL 


nsr'!'{no,  nl,  ne,^  ,  A^^;^,). 


Collecting  the  above  theorems,  we  reach  the  final  theorem  which  gives  the  mini- 
mum number  of  spatial  relations  of  a  line  string. 

Theorem  V.    For  a  line  string  {rc,Oc,rp,dp,ri,0i)  with  given  numbers  of  centroids 

NC>0,  objects  NO>0,  endpoints  NP>NO,  line  segments  0<NL<(^'^^'^^+^y  and 

equal  signs  0<NE<2NC+2NP+2NL-6,  where 

'  NO>0  if  NC  =  0 

<  NP  =  0  A  NC  >  0    ifNO  =  0, 
,  NC  =  NO  otherwise 

the  number  of  spatial  relations  NSR,s  of  the  line  string  {rc,0c,rp,6p,ri,6i)  is  given  by 
NSRu{NC,  NO,  NP,  NL,  NE) 


'  [j2NSR,iNC,NEr,,NEe^)] 

[^NSRp{NO,NP,NE,^,NEe^), 
ZNSR,{NC,NE^^,NEeJ 
G  <  J2^SRp  {NO,  NP,  NEr^ ,  NEe^)] 


if  NP  =  0 


if  NL  =  0 


[j:NSRp,  {NO,  NP,NL,  NE,^ ,  NE^^ ,  NE,, ,  NE,, ) , 
^NSR,{NC,NE,^,NEeJ 

^NSRp,  {NO,  NP,  NL,  NE,^,  NE,^,  NE,, ,  NE,,)]  otherwise 

where  the  sum,  denoted  by  E,  is  taken  over  all  nonnegative  integers  NE,^  <  NC  -  I, 
NEe^  <NC-l,  NE,^  <NP-l,  NEe^  <NP-l,  NE,,  <NL-l,  NEe,  <NL-1,  and 
E  NE^=  NE. 
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Figure  13  shows  iV5i?/,(0,2,3, 1,0)  >  E(ii7)(i|T)  E  (it)  (ft) 
of  a  line  string  with  NC=0,  N0=2,  NP=3,  NL=1,  and  NE=0. 


18  spatial  relations 
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Figure  13.  The  minimum  eighteen  spatial  relations  of  a 
line  string  with  A^C=0,  m=2,  NP=3,  NL=1,  and  NE=0. 


5.2  Space  Needs 


The  space  needs  of  an  image's  line  string  depend  on  the  image's  complexity,  the 
threshold  value  of  algorithm  APPROX_LINE,  and  the  objects'  configurations.  After  the 
linear  image  is  generated,  the  space  needs  of  its  line  string  can  be  found  as  follows. 
Assume  each  symbol  or  number  needs  one  byte  of  memory.  The  maximum  space  needs 
for  a  line  string  with  NC  centroids,  NP  endpoints,  NL  line  segments,  and  NE  equal  signs 
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are  2(NC+NP+NL)+NE+3  bytes.  Without  considering  the  equal  sign  '=',  a  line  string 
needs  NC  bytes  for  length  r^,  the  NC  bytes  for  angle  9^,  and  one  byte  for  the  delimiter 
between  and  6^  for  NC  centroids.  Therefore,  2NC+1  bytes  are  for  NC  centroids  in  a 
line  string  when  NC  >  0.  Under  the  same  reasoning,  NP  endpoints  need  2NP+}  bytes 
and  NL  lines  require  2NL+1  bytes.  Hence,  the  space  needs  are  2(NC+NP+NL)+NE+3 
bytes  since  NE  equal  signs  need  NE  bytes.  Figure  14  displays  the  relationship  between 
space  needs  and  numbers  of  spatial  relations  NSR  of  line  strings.  The  NSR  increases 
abruptly  because  it  is  in  0[ispace/n)l]  where  n  is  a  positive  integer.  These  numbers, 
created  by  projection,  are  conservative.  The  actual  numbers  should  be  much  higher  than 
these  projected  values.  This  graph  shows  the  maximum  space  of  16  bytes  and  lacks  the 
genuine  relationship  among  NSRt,  NSR^,  NSRp,  and  NSR,,,.  We  would  expect  NSRi^  to 
have  the  largest  value  when  space  exceeds  a  threshold  value,  which  would  not  be  large. 
Second  comes  NSRp,,  third  NSR,,  and  last  NSRp. 


Relationship  between  space  needs  and  NSR  of  line  strings 
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Figure  14.  Relationship  between  space  needs  and 
numbers  of  spatial  relations  NSR  of  line  strings. 


5.3  Comparative  Performance  Analysis 


This  section  augments  the  empirical  analyses  of  Chang  and  Lee  [8]  in  calculating 
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the  storage  space  and  matching  speed  of  line  strings.  Assume  the  image  object  has 
a  rectangular  shape.  Since  a  rectangular  object  in  a  line  string  needs  two  endpoints 
or  one  if  it  is  square,  the  maximum  number  of  symbols  for  rectangular  objects 
is  8N+3:  [A^(centroids)+2xA^(two  endpoints  for  an  object)+A^(one  line  segment  for  an 
object)]  x2(one  for  length  r  and  one  for  angle  (delimiters).  Table  1  shows  the  number 
of  symbols  required  among  various  string  image  data  structures.  The  storage  space  of 
8N+3  bytes  is  more  than  RCOS  string's  5N  bytes  for  object  symbols.  Line  strings, 
however,  provide  more  spatial  information  than  RCOS  strings. 


Table  1.  Storage  space  and  its  ratios  of  various  string  image  data  structures.  B:  2D 
B-string,  C:  2D  C-string,  G:  2D  G-string,  R:  RCOS  string,  and  L:  Line  string. 


Number 

Number  of  symbols 

of 

2D  B- 

2D  C- 

2D  G- 

RCOS 

Line 

L/B 

L/C 

L/G 

L/R 

objects 

string 

string 

string 

string 

string 

(N) 

2 

8.0 

7.6 

10.0 

10.0 

19.0 

2.38 

2.50 

1.90 

1.90 

4 

16.0 

19.6 

43.6 

20.0 

35.0 

2.19 

1.79 

0.80 

1.75 

8 

32.0 

57.2 

173.2 

40.0 

67.0 

2.09 

1.17 

0.39 

1.68 

16 

65.6 

164.4 

672.0 

80.0 

131.0 

2.00 

0.80 

0.19 

1.64 

32 

141.0 

476.8 

2673.2 

160.0 

259.0 

1.84 

0.54 

0.10 

1.62 

64 

291.4 

1422.0 

9626.0 

320.0 

515.0 

1.77 

0.36 

0.05 

1.61 

128 

635.6 

4768.8 

32479. 

640.0 

1027.0 

1.62 

0.22 

0.03 

1.60 

The  dispositions  of  two  rectangular  objects  can  create  169  types  of  spatial  relation- 
ships [8].  Table  2  contains  the  number  of  operations  required  to  retrieve  any  of  the  769 
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relations  for  three  string  image  data  structures. 


Table  2.  Computational  speed  in  retrieving  one  of  the  169  types  of  spatial  relations. 


Number  of  operations 

2D  B-string 

RCOS  string 

line  string 

No.  of  comparisons 

0 

6 

16 

No.  of  subtractions 

16 

0 

0 

No.  of  multiplications 

8 

0 

0 

Compared  to  other  string  image  data  structures,  the  drawback  of  line  strings  may 
arise  from  the  increased  need  for  storage  space  or  a  slow  down  in  matching  speed. 
Depending  on  a  user's  need,  the  above  statistics  demonstrate  that  the  effective  spatial 
representation  property  of  line  string  may  compensate  for  the  increase  in  space  or 
computational  speed. 


CHAPTER  6 

A  LONGEST  APPROXIMATE  COMMON  SUBSEQUENCE  PROBLEM 

Finding  a  longest  common  subsequence  of  two  strings  occurs  in  a  number  of 
computing  and  data-processing  applications.  A  classical  longest  common  subsequence 
problem  [21]  (abbreviated  LCS)  is,  given  two  strings  X  and  Y,  to  find  a  maximum  length 
common  subsequence  of  X  and  Y.  A  subsequence  of  a  given  string  is  just  the  given  string 
with  some  symbols  (possibly  none)  left  out.  String  Z  is  a  common  subsequence  of  X 
and  y  if  Z  is  a  subsequence  of  both  X  and  Y.  Finding  an  LCS  is  mainly  used  to  measure 
the  discrepancies  between  two  strings.  An  LCS,  however,  does  not  always  reveal  the 
degree  of  difference  between  two  strings  that  some  problems  require.  For  example,  if  Sg 
=  {a,  b),  s,  =  {b,  b)  and  S2  =  {b,  a),  an  LCS  (b)  of  Sq  and  s,  is  the  same  as  an  LCS 
of  So  and  S2.  From  the  viewpoint  of  LCS,  the  resemblance  of  Sj  and  Sg  is  the  same  as 
the  resemblance  of  S2  and  Sq.  However,  ^2  has  more  symbols  in  common  with  Sq  than 
s,  does,  although  not  in  the  same  order.  Approximating  an  LCS  may  better  characterize 
the  discrepancies  between  two  strings. 

6.1  Definition 

A  longest  approximate  common  subsequence  problem  (abbreviated  LACS)  produces 
a  maximum-gain  approximate  common  subsequence  of  two  strings.  An  approximate 
subsequence  of  a  string  X  is  a  string  edited  from  a  subsequence  of  X.  The  only  editing 
operation  allowed  here  is  an  adjacent  symbol  interchange.  String  Z  is  an  approximate 
common  subsequence  of  two  strings  X  and  y  if  Z  is  an  approximate  subsequence  of 
both  X  and  Y.  The  gain  function  g,  which  will  be  described  later,  assigns  a  nonnegative 
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real  number  to  each  subsequence.  Formally,  the  LACS  problem  is  defined  as  follows: 
Given  two  strings  X  and  Y,  a  weight  W„>0  for  a  symbol  in  an  approximate  common 
subsequence,  and  a  weight  W,<0  for  an  adjacent  symbol  interchange  operation,  a  string 
Z  is  a  longest  approximate  common  subsequence  of  X  and  7  if  Z  satisfies  the  following 
two  conditions: 

1)  Z  is  an  approximate  common  subsequence  of  X  and  Y,  and 

2)  the  gain  giX,Y,Z,WM  =  \Z\W^+SiX,Z)W,+ 

S(Y,Z)W,  is  maximum  among  all  approximate  common  subsequences  of  X  and 
Y,  where  6(X,Z)  is  the  minimum  edit  distance  from  a  subsequence  of  X  to  Z,  so 
is  6iY^  to  Y  and  Z. 

A  string  Z  is  said  to  be  of  edit  distance  A;  to  a  string  Z'  if  Z  can  be  transformed  to  be 
equal  to  Z'  with  a  minimum  sequence  of  k  adjacent  symbol  interchanges.  The  following 
is  an  LACS  example.  Let  X  =  {B,  A,  C,  E,A,B),Y=  (A,  C,  D,  B,  B,  A),W^  =  3,  and 
W,  =  -/.  A  longest  approximate  common  subsequence  of  X  and  Y  is  Z=  {A,  B,  C,  B,  A) 
with  the  gain  g(X,Y,Z,W„,W,)  =  mW„,+6(X,Z)WM(Y,Z)W,  =  Jx3+2x(-7)+7x(-7)  =  12. 

6.2  Computational  Complexity 

This  section  breaks  up  the  complication  of  the  LACS  by  sorting  the  problem  into 
LACS,,  0<  i  <  mm{\X\,\Y\}-J  categories,  where  X  and  Y  are  input  strings.  Subsequently, 
we  prove  that  LACS™„,i;(,,in)-y  problem  is  NP-hard  and  conjecture  that  the  problems  from 
LACS,  to  LACS^„,i;^||n).2  are  at  least  as  hard  as  the  NP-complete  problems. 

6.2.1  LACS  Categories 

The  LACS  problem  fits  into  categories  according  to  the  relation  between  the  weights 
W„  >  0  and  W,  <  0: 
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•  LACSo — when  0  <  W„  <  -W,„  LACSo  is  reduced  to  an  LCS  problem  since  no 
adjacent  symbol  interchanges  are  allowed  for  any  symbol  in  LACSo; 

•  LACS; — when  -W,  <  W„,  <  -2W„  any  symbol  in  LACS;  makes  no  more  than  1 
adjacent  symbol  interchange; 

•  LACS2 — when  -2W,  <  W„  <  -3W,,  any  symbol  in  LACS2  makes  no  more  than 
2  adjacent  symbol  interchanges; 

•  LACS; — when  -iWs  <W„<  -(i+])W„  any  symbol  in  LACS,  makes  no  more  than 
i  adjacent  symbol  interchanges;  and 

•  LACS  min|ixi,m)-/ — when  -(min{IXI,iy|}-7)W,  <  W„,  X  and  Y  are  input  strings,  any 
symbol  in  LACSn„„|i;^jn|.;  makes  no  more  than  min{IXl,IJ1}-7  adjacent  symbol 
interchanges,  which  is  the  maximum  number  of  interchanges  a  symbol  is  allowed 
to  make. 

Another  useful  abbreviation  is  that  an  LACSi{X,Y)  equals  an  LACS,  of  X  and  Y.  The 
LACS,  problem  can  be  interpreted  in  another  way  called  a  trace  [49].  Diagrammatically 
aligning  the  input  strings  X  and  Y  and  drawing  lines  from  symbols  in  X  to  their  matches  in 
y  provides  the  trace  of  X  and  Y.  Figure  15  illustrates  the  example  in  Section  6.1  through 
trace.  In  an  LACS,  trace,  each  line  is  allowed  to  have  a  maximum  of  /  line-crossings,  i.e. 
the  symbol  touched  by  the  line  may  make  no  more  than  /  adjacent  symbol  interchanges. 
The  total  number  of  line-crossings  in  a  trace  is  6{X,Z)  +  S{Y,Z). 


X  =  {B  A   C  E  A  B) 


Y  =  {A    C    D    B    B  A) 


UKCS2{X,Y)  =  Z=  {A  B  C  B  A) 
g  (X,Y,Z,3,-1)  =  5x3+3x(-1)  =  12 


Figure  15.  An  LACS2  illustrated  through  trace 


42 

6.2.2  The  LACSn^„,ixi,m)./  Problem  is  NP-hard 

In  Theorem  VI,  we  show  that  any  instance  of  extended  string-to-string  correction 
problem,  which  was  proven  to  be  an  NP-complete  problem  by  Wagner  in  1975  [48],  can 
be  reduced  in  polynomial  time  to  an  instance  of  LACSmi„(ixi,i>i)-/.  The  extended  string- 
to-string  correction  problem  (ESSCP) — given  finite  alphabet  S,  two  strings  X  and  Y  € 
S',  and  a  positive  integer  k — determines  whether  there  is  a  way  to  derive  the  string  Y 
from  the  string  X  by  a  sequence  of  k  or  fewer  operations  of  single  symbol  deletion  or 
adjacent  symbol  interchange. 

Theorem  VI  (LACS„i„{m^ty\}.i  is  NP-hard) 

If  X  and  Y  are  input  strings,  then  the  LACSmi„|ixi,i}i)./  problem  is  NP-hard. 
Proof  We  first  show  that  LACSnj„,ixi,m)./  does  not  belong  to  NP.  Given  an  instance 
of  the  problem,  we  use  as  a  certificate  an  LACSnu„,ixi,m)./  ZofX  and  Y.  A  verification 
algorithm  checks  if  the  gain  g{X,Y,Z,W„,W,)  =  mW„+SiX,Z)W,+6{Y,Z)W,  >  k',  a 
nonnegative  real  number.  From  the  ESSCP,  we  know  it  is  unlikely  to  find  S{X',Z) 
and  S{Y',Z)  in  polynomial  time,  where  X'  and  Y'  are  subsequences  of  X  and  Y, 
respectively.  If  8{X',Z)  and  8{Y',Z)  cannot  be  found  in  polynomial  time,  then  8{XJ) 
and  8{Y,Z)  definitely  cannot  be  found  in  polynomial  time.  Therefore,  we  could  say 
with  certainty  that  a  polynomial-time  verification  algorithm  does  not  exist. 

To  prove  that  LACS™„,:;o,m)-/  is  NP-hard,  we  show  that  ESSCP  <p  LACS^„,m,|„,.,. 
In  other  words,  any  instance  of  ESSCP  can  be  reduced  in  polynomial  time  to  an 
instance  of  LACS^„,yf,,in,./.  Let  the  two  input  strings  be  the  same  at  both  problems. 
We  now  show  that  string  X  needs  k  operations  of  deletion  or  interchange  to  derive 
string  Y  if  and  only  if  the  corresponding  hkCS^^^mmvii^^y)  problem  has  an  LACS 
Z  with  a  gain  g{X,Y:Z,WM  ^  \Y\W^+ik-\X\+\Y\)W,.  Suppose  that  string  X  needs  k 
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Operations  of  single  symbol  deletion  or  adjacent  symbol  interchange  to  derive  string 
Y.  The  number  of  deletions  has  to  be  IXI-IH,  making  the  number  of  interchanges 
A:-IXI+iyi.  Thus,  for  an  LACSn,in|ixi,m).y(X,y)  trace,  there  are  \Y\=\Z]  lines  (matchings) 
and  A:-IXl+iyi  line-crossings  (interchanges).  Therefore,  the  gain  g{X,Y,Z,W„,W,)  is 
iyiW„+(^-IXl+|}1)\V,.  Conversely,  suppose  that  Z  =  LACS™„(ixi,in)-/(X,y)  has  a  gain 
giX,Y,Z,W„,W,)  =  \Y\W„+{k-\X\+\Y\)Ws.  Because  X  has  \Y\  symbols  in  common  with 
Y,  Z  could  equal  Y.  X  then  needs  lAH-iyi  deletions  and  ^-IXI+iyi  interchanges  to  derive  Z, 
i.e.,  Y.  Thus,  for  the  ESSCP,  string  X  needs  \X\-\Y\  deletions  and  A:-iyi+IXI  interchanges 
to  derive  string  Y.  The  total  number  of  operations  required  to  derive  Y  from  X, 
therefore,  is  k  =  i\X]-\Y\)+{k-\Y\+\X\).  □ 


CHAPTER  7 
A  HEURISTIC  APPROXIMATION  ALGORITHM 


Despite  the  unlikelihood  of  finding  a  polynomial-time  algorithm  for  solving  the 
LACS  problem  exactly,  near-optimal  solutions  in  polynomial  time  may  still  be  possible. 

7.1  ALACS  Algorithm 

An  approximation  algorithm  finds  an  approximate  LACS  (abbreviated  ALACS)  of 
two  strings.  The  procedure  APPROX_LACS  repeatedly  calls  an  LCS-routine  [WmZ-Wv] 
times,  beginning  with  allowing  zero  number  of  edit  operations  for  each  selected  symbol. 
Each  round,  the  allowable  number  of  edit  operations  is  increased  by  one.  The  symbols 
selected  are  marked  off  from  the  input  strings  to  prevent  consideration  in  the  next  round. 
Suppose  X  and  Y  are  input  strings,  and  the  trace  T  is  empty  and  /  =  0  at  first.  It  executes 
the  following  steps. 

1.  Find  an  LCS  of  X  and  Y. 

2.  Select  symbols  from  the  LCS  such  that  each  symbol  makes  no  more  than  / 
line-crossings  in  trace  T. 

3.  Eliminate  the  selected  symbols  from  X  and  Y,  and  add  them  to  T. 

4.  /  =  /-!-  1. 

5.  Repeat  the  above  steps  until  /  >  [Wy-Vy,]. 

The  procedure  uses  two  arrays  m[O..IXI]  and  n[0..\Y\]  to  flag  which  symbols  in  X  and  Y 
are  selected.  If  a  symbol  is  selected,  it  stores  the  index  of  matching  symbol  of  the  other 
string.  If  the  symbol  is  not  selected,  it  stores  0.  An  LCS  function  LCS_LENGTH  is 
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borrowed  from  Gormen  et  al.  [14],  and  is  modified  to  include  checking  whether  symbols 
in  X  and  Y  are  selected. 

APPROX_LACS  (X,  Y,  W^,,  WJ 

/*  X  and  Y:  input  strings  */ 

/*  W„:  weight  for  a  symbol  in  an  approximate  common  subsequence  */ 
/*  W^:  weight  for  an  adjacent  common  symbol  interchange  operation  */ 

1  for  /i  ^  0  to  -  1 

2  do  for  /  ^  7  to  IXI 

3  do  mli]  <—  0 

4  for    ^  0  to  171 

5  do  n[j]  ^  0 

6  LCS_LENGTH  (X,  Y,  m,  n,  b) 

7  SELECT _SYM  (X.  Y,  \X\,  \Y\,  m,  n,  b,  h) 

Procedure  SELECT_SYM  finds  an  LCS  and  selects  symbols  from  it.  It  makes 
sure  the  selected  symbols  do  not  make  more  than  the  maximum  allowable  number  of 
edit  operations. 

SELECT_SYM  (X,  Y,  i,  j.  m,  n.  b,  h) 


1  if  i  =  0  or  j  =  0 

2  then  return 

3  if  b[i.j]  =  \ 

4  then  if  ~OVER_CROSSING(X.  Y  i,  j,  m,  n,  h) 

5  then  m[i]  <—  j 

6  n[j]  ^  i 

7  SELECT_SYM(X,  Y,  i-1,  j-J,  m,  n.  b.  h) 
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8  dseif  b[i,j]  =  'f 

9  then  SELECT_SYM(X,  Y,  i-1,  j,  m,  n,  b,  h) 

10  else  SELECT_SYM(X,  Y,  i,  j-],  m,  n,  b,  h) 

Procedure  OVER_CROSSING  checks  a  line  from  the  ith  symbol  of  X  to  the  yth 
symbol  of  Y  does  not  cross  more  than  h  other  lines  in  trace. 

OVER_CROSSING  (X,  Y  i,  j,  m,  n,  h) 

1  hi  =  0 

2  for  il      7  to  i-1 

3  do  if  m[il]  >  j 

4  then  hi  =  hi  +  1 

5  for  jl  ^  1  to  j-1 

6  do  if  n[jl]  >  i 

7  then  hi  =  hi  +  1 

8  if  hi  >  h 

9  then  return  TRUE 

10  else  return  FALSE 

For  this  approximation  algorithm,  the  running  time  is  Od'Wy-W'vKlXI+iyi)^)  and 
the  space  needs  is  O(LYIiyi)  for  the  LCS_LENGTH  function.  Figure  16  illustrates  the 
progress  of  the  approximation  algorithm  on  an  instance.   We  name  the  trace  T  after 
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X  =  (C   A    B    E   B  A) 

s 

Y  =  {B   A    D   A    C  B) 

ALACSo(X,Y)=  Z'  =  <C  6) 
g{XXZ".3r1)=  2x3=  6 


Y  =  {B   A    D   A    C  B) 

ALACS/(X,y)=  Z=  {A  C  B) 
g(X.Y.Z'.3.-1)=  3x3+1x(-1)  =  8 


E   B  A) 


X=  { 


Y  =  {B   A    D   A    C  B) 


ALACS2(X,y)=  Z=  {A  C  B  B) 
g{X.Y. 1,3,-1)  =  4x3+3x(-1)  =  9 


Figure  16.  A  sequence  of  ALACS0..2  produced  by  APPROX_LACS  on  an  instance. 


executing  tiie  ith  LCS-routine  as  an  ALACS,.;,  /</<min{IXl,iyi}.  The  ALACS,  means 
a  symbol  in  ALACS,  can  make  no  more  than  /  adjacent  symbol  interchanges.  Figure 
17  is  an  instance  of  an  LACS2  problem.  The  difference  between  the  optimal  gain  and 
approximate  gain  is  1. 


X  =  ^^^j^  ^ 

Y  =  {B    A    D    A    C  B) 


LACS2(X,y)=  Z=  (A  B  A  B) 
g  (X,Y,Z,3,-1)  =  4x3+2x(-1)=  10 


Figure  17.  An  LACSj 
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7.2  Ratio  Bound 

For  a  maximization  problem,  we  say  that  an  approximation  algorithm  for  the 
problem  has  a  ratio  bound  p{n)  if  for  any  input  of  size  n,  the  gain  G  of  the  solution 
produced  by  the  approximation  algorithm  is  within  a  factor  of  p{n)  of  the  gain  G'  of 
an  optimal  solution,  namely,  G7G  <  pin).  For  an  ALACS,(A:,F),  0<i<min{IXI,m}-7, 
problem,  the  worst  ratio  bound  is: 

max  gain  of  LAC  Si 
mm  gain  oj  ALACbi 

The  next  two  lemmas  show  how  to  compute  the  minimum  gain  of  ALACS,  and 
the  maximum  gain  of  LACS,,  respectively. 

Lemma  IV  (Minimum  gain  of  ALACS i) 

If  Z  is  an  LCS  of  strings  X  and  Y,  the  minimum  gain  of  an  ALACS,(X,}0, 
0<i<min{IXI,m}-/,  is  mw„  for  any  /. 

Proof  The  procedure  APPROX_LACS  is  implemented  by  calling  the  LCS-routine 
[Wy-W,]  times.  Each  time,  they  select  symbols  from  the  LCS,  add  them  to  the 
ALACS,  and  remove  them  from  input  strings.  Since  it  selects  every  symbols  from  the 
first  LCS,  the  minimum  gain  of  the  approximation  algorithm  is  \7\W„.  □ 

Before  discussing  Lemma  V,  some  terminologies  need  introduction.  A  set  of  lines 
is  completely-crossing  if  each  line  crosses  every  other  line  in  the  set.  If  a  set  of  lines  is 
independent,  then  every  line  in  the  set  does  not  cross  any  line  in  the  other  set. 

Lemma  V  (Maximum  gain  of  LACSJ 

If  Z  is  an  LCS  of  strings  X  and  Y,  the  maximum  gain  of  an  LACS,(X,y), 
(K/<min{IXI,m}-7,  is  (i+2)\Z\WJ2. 

Proof  We  show  that  an  LACS,  has  a  maximum  gain  when  there  are  IZl  independent. 
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completely-crossing  and  /+7-line  sets  in  its  trace.  The  requirement  can  be  broken 
into  four  conditions  which  are  examined  separately. 

1.  IZl  sets:  Every  independent  set  contributes  at  least  one  symbol  to  an  LCS.  If  the 
number  of  independent  sets  is  more  than  IZl,  then  the  length  of  the  LCS  of  X  and 
Y  is  longer  than  IZl.  It  contradicts  the  assumption  that  Z  is  an  LCS  of  X  and  Y. 
On  the  other  hand,  the  gain  is  not  maximum  if  the  number  of  independent  sets  is 
less  than  IZl  since  each  set  contributes  a  fixed  gain,  which  will  be  explained  later. 
Therefore,  the  number  of  independent  sets  is  IZl  when  LACS,  has  a  maximum 
gain. 

2.  Completely-crossing:  Suppose  one  independent  set  is  not  completely-crossing, 
then  there  must  have  some  lines  not  crossing  one  another,  i.e.,  the  lines  are 
parallel  in  the  sense  of  LCS.  Each  of  the  parallel  lines  contributes  one  symbol  to 
an  LCS.  Since  the  length  of  the  LCS  is  fixed,  an  LACS,  achieves  the  maximum 
total  gain  only  by  having  the  maximum  gain  from  each  line.  For  an  LACS,,  the 
maximum  gain  one  line  can  reach  is  when  the  set  of  this  line  and  other  /  lines  is 
completely-crossing.  Therefore,  each  set  has  to  be  completely-crossing  in  order 
to  have  the  maximum  gain  of  an  LACS,. 

3.  Independent:  Every  set  is  independent,  otherwise  it  is  not  completely-crossing. 

4.  /+ 7-line:  From  the  above  discussion,  we  know  each  set  is  completely-crossing. 
For  a  line  in  the  set,  it  is  only  allowed  to  have  no  more  than  /  line-crossings 
because  of  the  limitation  of  LACS,.  Therefore,  the  maximum  lines  (also  the 
maximum  gain)  a  set  can  own  is 

In  short,  there  are  IZl  sets  in  the  trace,  where  each  set  has  /+/  lines  and  1+2+  ...  +/  = 
i(i+l)/2  line-crossings,  when  an  LACS,  has  a  maximum  gain.  Consequently,  the  max- 
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imum  gain  of  an  LACS,  is  m[{i+i)W^+i(i+l )W/2]  <  ii+2)mWJ2  since  -iW,<W„<- 
(i+l)Ws.  □ 

Theorem  VII  gives  the  worst  ratio  bound  of  ALACS,  by  applying  the  above  two 
lemmas.  Since  we  take  a  very  conservative  approach  to  find  Piin),  the  actual  ratio  bound 
is  expected  to  be  much  less  than  (i+2)/2  when  /  >  io,  a  positive  constant. 

Theorem  VII  (The  worst  ratio  bound  of  ALACS i) 

The  worst  ratio  bound  /?,(«)  of  an  ALACS,(X,y),  0</<min{IXl,m}-7,  is  (i+2)/2  for 
any  input  size  n. 

Proof  From  Equation  30  and  Lemmas  I  and  II,  the  worst  ratio  bound  /?,(«)  of  an 
ALACS,(X,y),  0<j<min{IXI,m}-/,  is  [(i+2)\Z\WJ2]/(mWJ  =  (i+2)/2  no  matter  what 
the  input  size  n  is.  □ 


CHAPTER  8 
AN  OPTIMIZATION  NEURAL  NETWORK 


A  modified  Hopfield  neural  network  is  designed  to  solve  the  LACS  problem.  A 
technique  of  interception  is  used  to  determine  the  values  of  network  coefficients. 

8.1  The  Hopfield  Network 
Hopfield  [22]  discovered  a  Liapunov  function  as  the  energy  function  of  the  network: 


In  solving  an  LACS  problem,  this  energy  function  is  compared  with  another  function 
built  from  LACS  constraints  in  order  to  determine  the  network  weights.  Let  strings 
X  =  {xj,  X2,  .  .  .  ,  x„)  and  Y  =  {y,,  y2,  .  .  .  ,  be  an  instance  of  an  LACS  problem. 
The  Hopfield  net  involves  mn  units  represented  as  an  mxn  array.  The  energy  function 
constructed  from  the  LACS  problem  constraints  is 


A,  B  and  C  are  constants.  0,y,  with  J  <  i  <  m  and  1  <  j  <  n,  indicates  whether  x,  matches 
yj.  Functions  gij,  hU  and  h2j  are 


E  =  El  +  Es  +  E3 


where 


4      '        1     /=,  +  ;      ,'  =  ( 


/»1'  =  I^U/I^U,  and  h1,^\X\J\Y\ 
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where         is  the  number  of  symbol 


jc,  in  string  X.  E,  and  E2  reflect  the  constraints 


that  each  row  (X)  or  column  {Y)  contains  a  fraction  hi  or  h2  of  a  single  /.  E3  reflects 
the  constraint  that  the  minimum  number  of  line-crossings  is  favored.  By  comparing  this 
energy  function  with  the  Liapunov  function,  the  weight  and  the  external  input  are  given  by 


Some  results  are  not  valid  when  the  input  string  is  beyond  a  certain  length,  e.g. 
about  7.  The  validity  can  be  improved  by  changing  the  coefficient  values,  but  this  may 
have  the  undesirable  effect  of  sacrificing  the  gain.  Therefore,  the  validity  of  results  is 
checked  after  convergence,  and  two  actions  are  taken  to  preserve  the  validity.  When  a 
line  makes  more  than  [Wy-W;,]  -  1  line-crossings,  it  is  canceled;  and  when  a  symbol  is 
picked  more  than  once,  only  the  first  pick  is  accounted  for. 


It  has  been  observed  that  the  convergence  and  the  results  of  Hopfield  net  to  the 
LACS  problem  is  highly  dependent  upon  the  coefficients,  and  different  input  strings  may 
have  different  optimal  coefficient  values.  The  values  of  coefficients  A  and  B  relative  to  the 
value  of  C  affect  the  net,  i.e.  AlC  (or  BIQ  affects  the  net.  With  sufficiently  large  values 
for  A  and  B,  the  low-energy  states  will  represent  valid  results  and  the  maximum  number 
of  matchings,  while  a  large  value  for  C  ensures  a  minimum  number  of  line-crossings. 
The  threshold  value  9^  is  fixed.  Thus  only  the  value  of  coefficient  A  needs  to  be  decided. 
Figure  18  shows  a  typical  curve  of  gains  and  coefficient  values.  It  resembles  the  shape  of 


^xy,x'y' —       A(jxy9xy'^xx'  Bg^yQi'y^yy' 


xy 


where 


8.2  Coefficient  Values  Determination  by  Interception 
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a  trapezoid  with  the  top  slowing  decHning,  and  eventually  becoming  a  constant.  The  value 
of  coefficient  A  is  determined  by  the  interception  of  two  lines,  which  are  extrapolated 
from  the  two  sides  of  the  curve.  From  some  random  instance  experiment,  the  peak  of  the 
curve  occurs  at  about  A  =  10,  and  the  curve  becomes  constant  at  A  >  200.  So  A  =  7  and 
A  =  5  are  picked  for  deciding  the  line  on  the  left  hand  side  of  the  curve,  and  A  =  100  and 
A  =  200  are  picked  for  the  other  line.  It  turns  out  that  the  value  of  A  is  almost  always 
selected  correctly.  This  is  because  the  curve  shown  in  Figure  18  applies  to  most  strings. 


An  example  of  coefficient  valuas  determined  by  interception 
30 1  1  p  '  p  ^ 


_i  I  I  '  ■  

5  10  15  20  25  30 

Coefficient  values 


Figure  18.  An  example  of  a  coefficient  value  determined  by  interception. 


CHAPTER  9 
LACS  ANALYSIS  AND  APPLICATIONS 

A  comparative  performance  of  the  two  LACS  approximation  methods:  the  heuristic 
algorithm  and  the  Hopfield  neural  network  is  given  in  this  Chapter.  This  Chapter  also 
lists  some  possible  applications  of  the  LACS  method. 

9. 1  Comparative  Performance  of  Approximation  Methods 


Table  3  Gains  and  running  time  of  string  matching 
algorithms  with  W„=3  and  W,=-l  on  some  random  instances. 


ILACSI 

LCS 

ALACS 

Hopfield  NN 

LACS 

ILCSIx3 

gain 

time 

gain 

time 

gain 

1 

3 

3 

0.16 

3 

0.23 

3 

2 

3 

5 

0.16 

5 

0.23 

5 

3 

6 

8 

0.17 

8 

0.22 

8 

4 

9 

11 

0.15 

11 

0.21 

11 

5 

9 

10 

0.17 

12 

0.30 

13 

6 

12 

15 

0.16 

14 

0.36 

15 

7 

12 

16 

0.17 

11 

0.33 

16 

8 

18 

20 

0.18 

20 

0.64 

20 

9 

12 

13 

0.18 

13 

0.53 

22 

10 

24 

25 

0.16 

25 

0.89 

25 

Table  3  lists  the  gains  and  running  time  of  LCS,  ALACS,  Hopfield  net,  and  LACS 
methods  with  W^=3  and  W,=-7  on  some  random  instances.  Figure  19  draws  a  relation  of 
gains  and  running  time  of  LCS,  ALACS,  Hopfield  net,  and  LACS  methods  with  Wn=6 
and  Ws=-1  on  some  random  instances.  The  output  gains  of  ALACS  and  Hopfield  net 
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are  pretty  much  the  same,  though  the  Hopfield  net  is  a  little  bit  poorer.  The  Hopfield 
net  is  much  slower,  but  it  requires  less  memory.  The  relation  distance  between  ALACS 
(or  Hopfield  net),  and  also  the  distance  between  ALACS  (or  Hopfield  net)  and  LCS, 
increases  with  increasing  WJ-W,  or  ILACSI.  The  experimental  results  also  show  the 
ratio  bound  pi{n)  =  (i+2)/2  of  an  ALACS,  is  exaggerated. 

Gains  of  random  Instances  of  LACS  praUam  tor  Wm>6  and  Ws^l 

90 
SO 
«T0 
^60 


|20 
10 

2  4  6  8          10         12         14         16         18  20 

ILACSI 

Figure  19.  Gains  and  running  time  of  string  matching 
algorithms  with  W„=6  and  W,=-l  on  some  random  instances. 

9.2  Possible  LACS  Applications 


Kruskal  [32]  suggested  some  possible  application  categories  for  sequence  compar- 
ison, which  LACS  belongs  to. 

1.  Molecular  biology:  The  main  use  of  sequence  comparison  in  molecular  biology  is 
to  detect  and  characterize  the  homology  (i.e.,  correspondence)  between  two  or  more 
related  sequences.  [50]  Two  structures  are  said  to  be  homologous  if  they  have  a 
common  evolutionary  ancestor,  and  are  said  to  have  a  high  degree  of  homology  if 
the  differences  that  have  developed  between  them  are  relatively  minor. 

2.  Human  speech:  Consider  the  much-studied  problem  of  recognizing  an  isolated  word 
selected  from  a  limited  vocabulary.   A  very  common  approach  is  to  provide  the 


 LCS 

-  -  ALACS 


Hopfield  NN 
-  ■ LACS 
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recognizer  with  one  or  several  versions  of  each  vocabulary  word  in  sequence  form. 
To  recognize  an  unknown  word,  the  recognizer  converts  it  to  sequence  form,  compares 
it  to  many  stored  sequences,  and  selects  the  one  that  matches  best. [15] 

3.  Computer  science:  Sequence  comparison  has  been  used  in  many  computer  science 
applications.  [20]  String-editing  and  file-comparison  are  two  among  the  many 
applications.  String-editing  is  the  correction  of  human  errors  made  at  the  input 
stage.  The  file-comparison,  where  each  line  or  record  of  the  file  may  be  treated  as 
a  single  symbol,  is  now  in  routine  use. 

4.  Codes  and  error  control:  Strings  of  symbols  are  used  extensively  to  code  information 
for  transfer  over  a  noisy  channel  such  as  radio,  telegraph,  microwave  transmission, 
and  so  forth.  Since  noise  in  the  channel  introduces  error  into  the  received  signal, 
many  questions  arise  about  detection  and  correction  of  error.  Though  the  primary 
focus  of  coding  theory  has  always  been  on  substitution  errors  (in  which  the  wrong 
symbols  are  received),  it  is  also  true  that  insertion  and  deletion  errors  (in  which  too 
many  or  too  few  symbols  are  received)  have  long  been  studied  under  the  heading 
of  synchronization  error.  Levenshtein  [37]  presented  a  distance  function  that  is 
appropriate  in  the  presence  of  insertion  and  deletion  errors.  His  distance  function 
plays  a  major  role  in  sequence  comparison. 

5.  Bird  song:  Bird  song  is  a  complex,  learned  behavioral  pattern.  The  development  of 
bird  song  shows  parallels  with  language  acquisition  in  humans.  Sequence  comparison 
analysis  of  bird  songs  produces  a  useful  measure  of  distance  between  pairs  of  songs. 
[6] 

6.  Human  depth  perception:  The  human  visual  system  fuses  images  from  the  two  eyes 
during  binocular  depth  perception  is  a  form  of  sequence  comparison.  [27] 

7.  Some  other  discrete-time  applications: 
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a.  Geological  strata:  In  geology,  stratigraphic  sequences  can  be  obtained  from  drill 
holes  and  outcroppings.  Each  stratum  may  simply  be  classified  as  to  type,  or  the 
classification  may  be  supplemented  by  additional  information  about  thickness, 
geochemical  or  mineral  assay,  fossils,  etc.  "Correlation"  of  different  sequences 
is  frequently  necessary;  i.e.,  it  is  necessary  to  determine  which  strata  in  one  such 
sequence  correspond  geologically  to  which  strata  in  another.  [45] 

b.  Tree  rings:  Dendrochronology  refers  to  the  science  of  dating  based  on  tree  rings. 
[16]  It  requires  the  comparison  of  sequences  obtained  from  different  logs,  most 
notably  from  logs  of  the  bristle-cone  pine. 

c.  Varves:  Varves  are  annual  layers  of  sediment,  generally  clay,  in  which  it  is 
possible  to  count  the  years,  typically  due  to  variation  between  the  summer  and 
winter  sediment. [23]  Varve  chronology  requires  comparison  of  sequences  from 
different  cores  or  different  locations. 

d.  Text  collation:  Collation  of  different  versions  of  the  same  text  offers  another 
possible  application. 

Some  other  continuous-time  applications: 

a.  Handwriting:  Time-warping  can  be  used  in  computer  processing  handwritten 
material  such  as  signatures  and  line  drawings.[46]  Typically,  the  penpoint  is 
tracked  during  the  act  of  writing  or  drawing,  with  pressure  of  pen  on  paper  or 
height  of  pen  as  a  third  coordinate,  so  that  the  record  consists  of  the  penpoint 
trajectory  as  a  function  of  time. 

b.  Evoked  potential  waves:  Evoked  potentials  from  the  brain  ("brain  waves"  in 
response  to  a  stimulus)  are  known  to  vary  somewhat  in  timing  from  one  occasion 
to  another,  and  might  constitute  an  application  of  time-warping. 


CHAPTER  10 
A  PROTOTYPE  IMAGE  RETRIEVAL  SYSTEM 


To  realize  the  advantages  of  the  above  methods,  a  prototype  traffic 
sign  retrieval  and  browsing  system  is  available  on  the  World  Wide  Web,  at 
http://www.cise.ufl.edur wenchen/idb.  The  experiment  results  show  any  image  in 
the  system  can  be  precisely  retrieved  as  long  as  each  image  has  a  different  image  string. 
The  accuracy  of  image  retrieval  depends  on  the  query  details. 

10.1  System  Overview 
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Figure  20.  System  structure. 
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Figure  20  presents  the  system  structure.  A  query  string  need  not  match  every 
image  string  in  the  database.  An  image  string  has  to  pass  through  a  filter  before  it  is 
qualified  for  matching.  An  image  string  passes  through  the  filter  only  if  it  includes  all 
image  objects  specified  in  the  query  string.  The  system  interface  is  shown  in  Figure  21. 

A  Prototype  Traffic  Sign  Retrieval  System 
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Line  string  Query 

•  Hi  Polar  or  mi  Rectangular  system 

•  BiLCS  si  LACS  ^ALACS  or  ^  NN  search  method 

If  not  LCS,  then  Wit,:    andW^:   


•  CenUoid  suing  {Xc.  Yc)  or  {Rc,  Ac): 
Endpoint  string  (Xp,  Yy)  or  {Rf^,  Ap): 
Line  string  (Xi,  Yi)  or  (Ri,  Ai): 

mi  Send    mm  Clear    mi  Help 
Figure  21.  System  interface. 

The  interface  provides  the  following  choices: 
1.   Polar  or  rectangular  coordination  system; 
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2.  LCS,  LACS,  ALACS,  or  Hopfield  neural  network  search  method;  and 

3.  Centroid,  endpoint,  or  Hne  string. 

So  far  the  system  is  not  able  to  handle  equal  signs  in  strings  since  the  search  methods 
do  not  cover  them.  If  a  user  chooses  the  LACS  search  mediod,  a  lengthy  or  indefinite 
waiting  time  might  occur,  because  the  LACS  method  is  not  a  polynomial-time  algorithm. 
The  number  of  elements  in  an  input  string  for  the  Hopfield  neural  network  can  not  exceed 
a  certain  number,  about  32.  The  number  of  elements  is  restricted  due  to  the  memory 
limitation.  The  Hopfield  net  involves  mn  units  represented  as  an  mxn  array,  where  m 
and  n  are  the  lengths  of  a  query  string  and  an  image  string,  respectively. 

10.2  Sample  Images 
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Figure  22.  Some  traffic  sign  samples. 


Figure  22  lists  some  traffic  signs  stored  in  the  system.  The  system  processes  276 
signs  among  the  hundreds  of  possible  traffic  signs.  The  other  traffic  signs  are  either 
too  complicated  to  be  segmented  or  contain  text  only  which  need  not  be  tested  again. 


Figure  22  shows  a  traffic  sign  and  its  skeletal,  linear,  and  centroidal  images.  It  also 


Skeletal  Image 

Linear  Image        Ceatroidal  Image 

S  T  0  P 

STOP 

■          ■          k  U 

Figure  23.  A  stop  sign  and  its  related  images. 


Table  4  Image  strings  of  Figure  23  stored  in  the  database. 


(Xe,  Y,) 

(  S  T  octagon  OP,  0  =  T  =  P  =  octagon  =  S  ) 

(Xp,  Yp) 

(S  =  SSSSSTT  =  TTOOOOOOOP  =  PPPPP, 
0  =  ST  =  PS  =  OOSP  =  POOPSSPOO  =  ST  =  P  =  T  =  T) 

(X„  Y,) 

(SPSOS  =  OOOPTPOSOOSPPT, 
T  =  POP  =  SO  =  SOPSSPO  =  OOOST  =  P) 

(re  ©c) 

(  S  T  octagon  0  P,  P  0  octagon  T  S  ) 

(rp,  Op) 

(SSSTSSSOOTOOTTOPOOPPPPP, 
POOPPPPOOPTOOOTTSSTSSSS) 

(r„  e.) 

(OPSSOSOPPSPOTOOOSTP, 
T=PPOSOSOPTOPOSOSSPO) 

illustrates  the  uncertain  property  of  the  curve-to-line  transformation.  Letter  'T'  has  a 
perfect  transformation  of  two  line  segments.  For  the  letters  'S',  'O'  and  'P',  users  have 
no  idea  precisely  how  many  lines  or  what  kind  of  lines  would  be  created  from  these  letters. 
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The  experiment  shows  the  fuzzy  computation  doesn't  hurt  the  system  performance  much, 
since  the  system  also  uses  fuzzy  algorithms  to  search  for  the  best  matches.  The  image 
strings  of  a  stop  sign  stored  in  the  database  are  shown  in  Table  4.  The  traffic  signs 
themselves  are  stored  in  an  image  store.  The  endpoint  and  line  strings  do  not  include  the 
octagon  frame.  The  frame  does  not  contribute  many  distinct  features,  but  instead  makes 
the  strings  look  more  complicated. 


Skeletal  Image 

Linear  Image 

Centroidal  Image 

-1 

i 

Figure  24.  A  traffic  sign  and  its  related  images. 
Table  5  Image  strings  of  Figure  24  stored  in  the  database. 


-4 

(Xc,  YJ 

(  arrow  rectangle,  rectangle  arrow  ) 

(Xp,  Yp) 

(  arrow  arrow_head  arrow_head  arrow_head  arrow_head  arrow 
arrow_head  =  arrow  =  arrow  arrow_head  arrow_head, 
arrow  arrow  arrow_head  arrow_head  arrow  arrow_head  arrow_head 
arrow_head  arrow_head  arrow_head  arrow  ) 

(X„  Y,) 

(  arrow  arrow_head  arrow_head  arrow  arrow_head  arrow_head, 
arrow_head  arrow_head  arrow_head  arrow_head  arrow  arrow  ) 
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Table  5  (Continued)     Image  strings  of  Figure  24  stored  in  the  database. 


(rc,  Oe) 

(  arrow  rectangle,  rectangle  arrow  ) 

(rp,  ep) 

(  arrow_head  arrow_head  arrow  arrow  arrow  arrow_head  arrow_head 

arrow_head  arrow_head  arrow_head  arrow, 
arrow  arrow  arrow_head  arrow_head  arrow_head  arrow_head  arrow 
arrow_head  arrow_head  arrow_head  arrow  ) 

(r„  0,) 

(  arrow_head  arrow_head  arrow_head  arrow_head  arrow  arrow, 
arrow_head  arrow_head  arrow  arrow_head  arrow  arrow_head  ) 

Figure  24  is  another  traffic  sign  with  one  straight  arrow  and  one  left  arrow.  Table 
5  lists  the  image  strings  of  Figure  24.  To  distinguish  between  signs  which  have  identical 
objects,  the  system  might  divide  an  object  into  several  parts.  For  example,  the  arrow 
here  is  divided  into  two  parts:  arrow  and  arrow_head.  For  the  reason  of  flexibility,  an 
object  might  be  stored  in  the  system  with  several  aliases,  e.g.,  a  car  could  be  an  auto, 
an  automobile,  a  vehicle,  .  .  .  ,  etc. 


10.3  Sample  Queries 


Traffic  Signs  with  Arrows 


@9SI 

1  EKtT 

L5J 

LLl 

Q 

1 

D 

1 

Q 

1 

OitLV 

/* 

DHLY 

t 

ONLY 

T 

m 

ONLY 

*»• 

'Mr\ 

••y 

<l> 

> 

•  ». 

a*"    /  ^ 
"  •  %^ 

Figure  25.  Traffic  signs  with  arrows. 
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Figure  25  lists  fifty-one  traffic  signs  with  different  kinds  of  arrows  in  this  system. 
To  retrieve  one  traffic  sign  with  a  specific  arrow  is  the  best  way  to  show  the  strength 
of  this  system.  Some  permutations  of  the  substrings  in  Table  5  will  be  used  in  the 
following  queries  to  retrieve  the  image  of  Figure  24.  Figure  26  is  a  search  results  list 
for  a  query  specifying  the  rectangular  system,  the  neural  network  search  method,  and  a 
centroid  string:  (  arrow  rectangle,  rectangle  arrow).  The  list  includes  images  with  arrows 
and  rectangular  frames.  The  results  are  not  satisfied,  since  a  centroid  string  provides 
little  information  for  the  system  to  use. 


Query  Results  List 


Caiti=l2    Gaiti=l2    Gaiti=l2    □aiti=l2    Gain=l2    Gain=l2    Gain=l2    Gain=l2  Gaih=l2 


OHVf 


mem 


Caih=l2 


Caiti=l2 


Gaih=l2 


Gain=12 


Caih=l2 


Caih=ll  Gaih=n 


Gaih=ll 


Ciih=U 
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ONLY 


Gaiti=ll    Gain=ll    Ga^h=U    Ga^ti=ll    Gain=U    Gain=ll    Ga<n=ll    Gaih=ll  □aih=ll 


Query 

(poUl-  system,  cetiUoid  sllihg  and  heuLal  ticlwcjllr  inethcxi  with  Wiin=4-  and  Ws=- 1) 
 C  arrow  ccctangLc,  rectangle  arcow  ) 


Figure  26.  Search  results  by  using  a  centroid  string. 


The  next  two  figures  demonstrate  that  the  ALACS  method  outperforms  the  LCS 
method.  Figure  27  is  the  search  results  list  for  a  query  specifying  the  rectangular  system, 
the  LCS  method,  and  a  line  string.  Figure  28  is  the  results  list  for  the  same  query,  shown 
in  Figure  27,  except  with  the  ALACS  method.  Both  results  show  the  desired  target  image, 
given  in  Figure  24,  is  located  in  the  second  category  of  gain.  The  ALACS  method  is 
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shown  to  outperform  the  LCS  method,  since  it  has  fewer  elements  at  that  category. 


Query  Results  List 
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Figure  27.  Search  results  by  using  the  LCS  method. 


Query  Results  List 
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[  arrow_hcadarrow_hcad  arrow  arrow  arrow_hcad  arrow  arrow, 
arrow  aero w_hcad  arrow  ) 

Figure  28.  Search  results  by  using  the  ALACS  method. 


The  last  two  search  results  will  show  the  differences  between  the  rectangular  and 
polar  coordination  systems  used  in  the  proposed  system.  The  search  results  for  queries 
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specifying  the  rectangular  and  the  polar  systems  are  illustrated  in  Figure  29  and  30, 
respectively.  Each  query  string  is  created  from  the  second  half  of  its  respective  line 
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Figure  29.  Search  results  by  using  the  rectangular  system. 
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Figure  30.  Search  results  by  using  the  polar  system. 


string.  Since  there  is  no  permutation  operation  on  the  strings,  the  search  results  are 
not  different  between  the  LCS  and  ALACS  methods.  The  query  with  the  polar  system 


67 

successfully  retrieves  the  desired  traffic  sign,  but  the  query  with  rectangular  system  does 
not.  Both  the  length  and  angle  data,  given  by  the  polar  system,  and  the  x  and  y  coordinates 
data,  given  by  the  rectangular  system,  supply  different  and  useful  information.  It  is  an 
advantage  for  the  system  to  provide  both  coordination  systems. 


CHAPTER  11 
CONCLUSIONS 


None  of  the  previous  image  representations  or  image  matching  methods  has  satisfied 
fully  the  two  major  demands  of  an  image  database:  (i)  accurate  and  fast  retrieval  and  (ii) 
efficient  storage.  This  study  attempts  to  meet  the  above  two  needs  through  a  line-string 
image  representation  and  through  a  new  image  matching  method.  The  representation 
better  characterizes  objects  yet  preserves  an  object-oriented  property.  The  technique 
of  finding  a  longest  approximate  common  subsequence  improves  the  image  retrieval 
methods.  However,  assuring  all  the  needs  of  an  image  database  remains  an  elusive  goal. 
Conclusions  of  these  works  and  possible  future  tasks  are  discussed  individually  on  each 
research  area. 

11.1  A  Line-String  Image  Representation 

To  utilize  fully  the  system's  superiority,  system  users  need  to  be  familiar  with  the 
image  skeletonizing  technique.  The  Zhang-Suen  transform  repeatedly  removes  boundary 
points  from  a  region  in  a  binary  image  until  an  irreducible  skeleton  remains.  Some 
special  cases  should  be  noted,  e.g.,  the  skeleton  of  a  disk  is  a  dot.  The  outcome 
from  APPROX_LINE  algorithm  applied  on  a  curve  is  uncertain.  Therefore,  images 
are  represented  approximately  by  line  strings.  This  makes  the  system's  performance 
depend  on  the  strength  of  the  fuzzy  matching  algorithm.  Inferior  descriptions  of  objects' 
relative  positions,  forms,  and  sizes  are  disadvantages  for  all  string  image  representations. 
Chain  code  [18,  19]  and  polygonal  approximation  [44,  40,  3]  representations  are  worthy 
candidates  for  alternate  image  representation  approaches. 
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1 1 .2  A  Line-String  Matciiing  Method 

This  study  is  a  preHminary  look  at  the  LACS  approximation  problem,  but  several 
open  questions  need  answers  before  it  becomes  definitive.  Thus  far,  only  the  last  case  of 
LACSi(X,Y),  7</<min{lX1,iyi}-7,  has  been  proven  to  be  NP-hard.  The  other  cases  remain 
open,  but  it  is  likely  that  the  vertex-cover  problem  [28]  may  be  reduced  in  NP-hard  proofs 
for  these  cases.  For  the  LACS  heuristic  approximation  algorithm,  an  enduring  difficulty 
is  the  worst  ratio  bound.  Either  it  must  prove  to  have  a  good  average  ratio  bound,  or 
another  good  ratio  bound  algorithm  must  be  designed.  Presently,  only  the  worst  ratio 
bound  exists  for  the  algorithm,  which  is  (/+2)/2  for  ALACS,.  It  is  believed  that  the  actual 
ratio  bound  is  a  much  smaller  number.  The  performance  of  LACS  Hopfield-style  net  is 
slightly  worse  than  the  heuristic  algorithm.  This  is  due  to  the  problem  of  local  minima. 
Adding  noise  terms  to  the  net  input  of  each  neuron  is  being  investigated  as  a  way  to  avoid 
this  problem.  The  LACS  definition  does  not  cover  the  equal  signs  in  strings.  Including 
equal  signs  in  strings  would  create  at  least  an  NP-hard  problem  for  the  matching  methods. 

11.3  An  Image  Retrieval  System 

A  traffic  sign  retrieval  system  has  been  developed  based  on  an  efficient  line-string 
image  data  structure  and  an  effective  line-string  matching  method.  Preliminary  results 
on  the  traffic  sign  image  database  show  that  this  is  a  promising  technique  for  content- 
based  image  database  retrieval.  There  are  a  number  of  research  issues  which  need  to 
be  addressed  to  improve  the  system  performance.  Simple  images  like  the  traffic  signs 
are  the  images  stored  in  the  system.  The  proposed  image  data  structure  and  image 
matching  method  are  capable  of  handling  complicated  images.  The  problem  lays  on 
image  segmentation  algorithms,  which  fail  on  complex  images.  A  future  extension  may 
be  in  the  direction  of  allowing  hand-drawn  sketch  query,  which  matches  objects'  sketches. 


70 

Edge  detection  algorithms,  which  are  used  to  find  objects'  sketches,  are  more  robust  than 
segmentation  algorithms. 


A  Prototype  Traffic  Sign  Retrieval  System 


Line  String  Query 
•  mi  Pblar  or  mj  Rectangular  system 


•  siLCS  -iLACS  ^  ALACS  or  ^  NN  search  method 

If  not  LCS,  then  Wir,:    and  Wsi   

•  Centroid  string  (JVc,  Yc)oi:{Rc,  Ac):  \ 


Endpoint  string  {Xp,  Yp)  or  (Rp,  Ap): 


Line  string  {Xi,  Yi)  or  {Ri,  Aj): 


si  Send     si  Clear    si  Help 


APPENDIX 
PROOFS  OF  THEOREMS 


Proof  of  Lemma  I:  Knuth  shows  the  number  of  ways  to  partition  a  set  of  n 
elements  into  m  nonempty  disjoint  subsets  is  the  Stirling  number  of  the  second  kind 
{^}.[30]  For  a  set  of  NC  centroids  with  NE  equal  signs,  there  are  NC-NE  nonempty 
sets,  {centroids  separated  by  '='s},  in  a  partition.  Therefore,  the  number  of  ways 
to  partition  a  set  of  NC  centroids  with  NE  equal  signs  into  NC-NE  nonempty  sets 
is  { 

NC-NE 

}.  Each  set  in  a  partition  is  either  a  single  centroid  set  or  an  '='  sep- 
arates every  two  centroids  in  that  set.  For  a  partition,  the  number  of  set  orderings 
is  (NC-NE)\.  Taking  into  account  the  number  of  ways  of  partition  and  the  num- 
ber of  set  orderings,  this  shows  that  the  number  of  spatial  relations  of  r<-  or  9^  is 

{NC-NE}iNC-NE)\.  D 

Proof  of  Lemma  II:  For  NO  objects  and  NP  endpoints,  the  multiset  of  NP 
endpoints  can  be  partitioned  into  various  nonempty  endpoint  multisets  5a={ object  /I's 

NO 

endpoints},  l<h<NO  such  that  l+J  Sh.=  U  =  multiset  of  all  endpoints  and  \U\=NP.  One 

h  =  l 

NO 

above  collection  ji)  5/^  with  NE  equal  signs  can  be  further  partitioned  into  NP-NE 

NO 

nonempty  multisets,  {endpoints  separated  by  '='s}.  Consequently,  the  collection  1+)  Sh. 

with  NE  equal  signs  can  be  partitioned  into  nonempty  multisets  n,x7;={ endpoints  sep- 

arated  by  '='s}  with  n,>0  and  l<i<I  for  which  l+ll^"'  =  l+l  Sh  and  E  NP-NE. 

'=>  h=i  1=1 

Without  considering  repetitive  multisets,  the  number  of  these  multiset  orderings  is  (NP- 
NE)\.  Summing  up  the  two  partitionings  and  the  number  of  orderings,  we  now  have 
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^  ;  {NP-NE)\  spatial  relations. 

To  count  out  the  duplication  of  these  repetitive  multisets,  the  above  result  should 
be  divided  by  11  n,!  since  there  are  n,  repetitions  for  each  multiset  T,.  This  shows  that: 

NSR'p{NO,NP,NE) 

=         ^        NSB!^{N0,NP,NE),  and 

S^=U,\U\=NP 

nsrI{no,np,ne) 

J2       {NP  -  NE)\ 

/  NO  7 

.^:^r"=.^,^.   n  n,! 


□ 


Proof  of  Theorem  II:  The  number  of  spatial  relations  of  (r^.^p)  is  equal  to 
the  product  of  numbers  of  spatial  relations  of  r,,  and  9p  minus  the  illegal  formations 
created  from  the  product.  The  illegal  formation  exists  when  one  endpoint  of  an  object 
overlays  another  endpoint  of  the  same  object,  i.e.,  they  have  the  same  coordinates. 
Therefore,  when  the  statement  i\ShnTi\  >  \\Sh\/2])  A  {\ShnRj\  >  [15^1/2])  is  true,  then 

^         ILL'piNO,NP,NEr,NEg)>  0.  l^/.nT.I  >  [|5;,|/2]  means  more  than  half 

of  object  /I's  endpoints  are  having  the  same  length;  \ShnRj\  >  [15^1/2]  represents  more 

than  half  of  object  /I's  endpoints  are  having  the  same  angle;  and  joining  both  makes  at 

least  two  endpoints  of  object  h  be  superimposing  on  each  other.  E  ra,=iVP  -  NEr  since 

1=1 

NO 

there  are  NP-NE,  multisets,  {endpoints  in  r,,  separated  by  '='s},  in  a  partition  of  1+)  5^, 

h  =  l 

J 

so  is  ^  mj=  NP  -  NEe  to  Op.  □ 


Proof  of  Lemma  III:    The  proof  is  similar  to  the  proof  of  Lemma  11  except  that 
the  line  segment  multisets  r„  l<i<NO  must  conform  to  the  following  conditions: 
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1.  r,={object  /'s  line  segments},  l<i<NO  for  which  0<ir,l<^' 2'')  since  the  maximum 
number  of  connected  lines  for  IS,!  points  is  ^'2'')'[12]  and 

NO 

2.  1+)  Ti=  V  =  multiset  of  all  line  segments  and  IV1=A^L. 
1=1 

In  addition,  Y,  mj=  NL  -  NE  since  there  are  NL-LE  multisets,  {line  segments  separated 

NO 

by  '='s},  in  a  partition  of  l+l  Tj.  □ 

1=1 

Proof  of  Theorem  III:    Similar  to  Lemma  HI  with  Theorem  n.  □ 

Proof  of  Theorem  V:  This  formula  just  collects  of  the  above  theorems.  The 
lower  bound  of  NSR„  is  ^NSRp  or  ^NSR,,,  when  NC>0  and  NP>0.  The  upper 
bound  of  NSRis  is  ^^NSRcXNSRp,  or  T,NSRcXNSRp  because  the  products  may  create 
some  bizarre  arrangements,  e.g.,  when  NC=2  and  NP=2,  the  number  of  spatial  re- 
lations is  4,  not  16  because  the  centroids  coincide  with  the  endpoints  in  this  case. 
In  the  future,  we  will  try  to  find  the  equation  of  NSRis,  not  just  its  lower  or  upper 
bounds.  □ 
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